If using chicken breasts, preheat your oven to 350°F. Place the chicken on a greased baking sheet, drizzle with oil, and season generously with salt and pepper. Bake for approximately 45 minutes until the internal temperature reaches 165°F. If using a whole chicken, place it in your large stock pot with celery, quartered onion, chicken base, salt, and pepper. Pour in enough water or broth to just cover the chicken (about 12 cups). Bring to a low boil over medium heat and cook for about an hour until the meat falls off the bone easily.
Once your chicken is fully cooked, remove it from the oven or pot and set it aside to cool slightly. If you boiled the whole chicken, strain out the celery and onion pieces from the broth using tongs or a fine mesh sieve, keeping all that flavorful liquid. When the chicken is cool enough to handle, shred it into bite-sized pieces using two forks. Discard any bones and skin.
While your broth is warming on the stovetop over medium heat, prepare the dumpling dough. In a medium bowl, whisk together the flour and baking powder. Add the cold cubed butter and work it into the flour using your fingers, a fork, or a pastry cutter until the mixture resembles tiny pebbles or coarse crumbs.
Pour the ice cold water or milk into the flour mixture and stir with a fork until everything comes together into a shaggy dough. Don't overmix - it should look rough and uneven at this stage. Use your hands to gently work the dough until it holds together in a ball.
Generously flour your work surface and turn the dough out onto it. Dust the top of the dough with more flour. Using a rolling pin, roll the dough out to about 1/8 to 1/4 inch thickness, rotating it clockwise periodically to prevent sticking. Add more flour under or on top as needed to keep things moving smoothly.
Using a pizza cutter or sharp knife, cut the rolled dough into strips about 1 inch wide, then cut across to make rectangles roughly 2 inches long. You can make them squares if you prefer - just keep them fairly uniform in size so they cook evenly. Dust the cut dumplings with a bit more flour and let them rest while you finish the broth.
Taste your broth and adjust the seasoning - this is crucial! Add salt and pepper gradually until it tastes really good on its own. You may need up to an additional tablespoon of salt depending on your broth. If you want a creamier texture, whisk in the cream of chicken soup now. For a thicker broth, mix the cornstarch with 1/4 cup cool water to create a slurry, then whisk it into the boiling broth.
Bring your seasoned broth up to a rolling boil. Now comes the important part: add your dumplings one at a time, dropping them into different spots in the pot so they don't stick together. Stir gently after adding several dumplings to keep them separated. Continue until all dumplings are in the pot.
Let the dumplings cook in the boiling broth for 15-20 minutes, stirring occasionally. You'll notice the broth starting to thicken from the flour on the dumplings. The dumplings will puff up, become tender, and some may sink to the bottom as they absorb the delicious broth. Test one by taking it out and tasting - it shouldn't taste doughy or raw.
Once the dumplings are cooked through, gently stir in your shredded chicken. Let everything simmer together for just a few minutes until the chicken is heated through. Taste one more time and adjust seasonings if needed. Serve this comforting dish hot in deep bowls, making sure everyone gets plenty of dumplings, chicken, and that wonderful broth.