Okay, I want to jump in here with some thoughts;
First of all, I think many of you guys are so fixated on reality that you miss the whole point of "Alternate History" especially pertaining to questions. I know I am new to this forum, but not to Alt-history.
I think what the O.P. is wanting to do is get some people thinking about how piston engines might have pushed the limits of technology for WHATEVER reason. Make a reason up, or just POOF jet turbines cannot exist because the universe says so.
The REASON is irrelevant, the idea is "HOW FAR COULD PISTON ENGINES HAVE GONE IF JET ENGINES NEVER HAPPENED?".
Or; "HOW FAR COULD PISTON ENGINES HAVE TAKEN AVIATION IF THEY REMAINED THE ONLY FORM OF TECHNOLOGY AVAILABLE."
This does not limit piston engines to not having superchargers, I think really, it just eliminates the turbine. In reality, the turbine was inevitable, because Steam turbines existed, it was only a matter of time before gas turbines also existed. But lets say turbines never exist because reasons. This does however restrict supercharging to mechanical only. Or, just allow turbos anyway. I suggest we allow turbos anyway since they don't require mechanical transmissions to change gear at various altitudes.
.....................
The first thing is that, like mentioned earlier, piston engine aircraft reach the point of diminishing returns around 5000hp for a single engine. With a sharp decline supposedly at about 7500hp for a single engine.
Looking at the pace and direction of engine development through the war, you get the idea that they could have reached the 5000hp limit by 1946 and the 7500hp limit by 1948. Air-cooled radial engines would find their limit around 5000hp just because you cant cool more than 4 rows deep in any meaningful way. Inline air-cooled engines could use cowling design to more effectively cool more rows of cylinders, drawing in air separately for rear-most or even individual cylinders; but radials reach a realistic limit at 4 rows without secondary ventilation.
Lycoming ran the XR-7755 in 1944; a liquid cooled radial consisting of 9 banks of 4 cylinders each, ad displacing 7,756cu-in, the 6,000lb engine made 5,000hp for takeoff/military power and 4,000hp for cruise. With a diameter of just 61 inches, it was quite compact.
The Pratt & Whitney R-4360-51VDT was 61 inches diameter, displaced 4362 cu-in, weighed 3,720lb and produced 4,300hp.
The Napier Sabre with a width of 40 inches, a height of 46 inches, displacing 2240cu-in and weighing in at 2360lb, started production with an already impressive 2200hp and had final prototypes making 3500hp by the end of development.
The Yakovlev M-501 was a 8760cu-in, 7-bank, liquid cooled radial with 6-cylinders per bank. It weighed 6500lbs with no turbo and 7500lbs with a turbo. Without a turbo it produced 4750hp and with a turbo it produced 6205hp. Oh, and did I mention it was a diesel? The turbocharger alone produced 551lb-ft of thrust. (Although, the engine was very good, it ended up powering ships since gas turbines happened.)
The designer felt the engine could produce 7000hp if fully developed. And proving him more than right, a modern example was modified for tractor-pulling and produces over 10,000hp.
The Napier Nomad managed to make 3150hp and weighing 3580lbs while being 56 inches wide and 40 inches high, it also produced 320lbf thrust from the jet portion of the engine.
The Napier Nomad gets dangerously close to being a jet engine in it's efforts to produce massive power with great fuel efficiency, basically it unknowingly becomes a motorjet but uses most of the power to drive a propeller like a turboprop rather than making thrust via ducted thrust (with or without expanding gasses; ducted fans are simple motorjets), as with a typical motorjet. Since it IS a motorjet by definition, I am allowing it. Maybe bending the rules a little, since it does have a turbine section that recoups power, but, since the turbine section operated as a turbocharger, (89psi boost!!!) it just barely fits I think.
((Aviation kinda forgot about the diesel piston engine and is worse off for it. With 30-60% less fuel consumption and only 30% more mass in early designs, it stands to reason that further development could have yielded some fantastic engines. Particularly since diesels seem to take increases in boost pressure and convert it to pure power without loss of cruise efficiency, I think that 2-stroke, turbo-diesel piston engines would likely have been the direction engine designers would have had to turn eventually to get both the power and efficiency needed to keep gaining ground.))
There are more examples that could be cited, but I think the point we see is best shown with the two US fighters. The P-47 and F4U. Both originally built with 1800hp, and later 2300hp R-2800 radials. Both had been prototyped with the new R-4360 engine. The F2G "Super Corsair" getting the "-4" variant with 3000hp and the XP-72 getting the "-13" variant with 3500hp. Both eventually could have been fitted with the final "-51" variant with made 4300hp. It should be assumed that if development continued, it is claimed (can't find the source) that the engine could possibly have been tuned up to 4700hp before a completely new, larger engine would need to be built.
The need for not only counter-rotating props but props with 5, 8, 15 blades would also be needed to both keep tip speeds as low as possible and bite enough air to put that much power into play. We see this development later (1960s-1990s) with the prop-fan, being driven by turbines producing more than 10,000hp, We also see development of ducted fans being driven by turbines with upwards up 100,000hp, in our hypothetical world they would be driven by piston engines. Of course, our piston engines will never make that much power. So, it would stand that aircraft would need more, smaller engines or aircraft would be smaller or both. However, breaking the sound barrier without the use of rockets is theoretically possible using motorjets.
While the realm of the supersonic would initially fall to rocket powered or at least rocket boosted aircraft, the motorjet could be developed to make enough thrust to push an aircraft past the sound barrier. Lets look at motorjets a bit;
In 1910 Henry Coanda made a motorjet with "cold" thrust. He later claimed that he intended to add an afterburner, and then later claimed that he HAD an afterburner and that the afterburner is why he crashed, it produced so much thrust that he stalled and crashed just above the runway. The drawings he used to prove his claim was reworked from the original copies and his claims are understood to be dubious at best. He claimed a thrust of 240lbf but modern engineers have calculated it to be 36lbf without the afterburner and 54lbf would be required to fly out of ground affect. He likely flew within ground affect, pulled up to fly and simply stalled and crashed. So, enough of his design, it was never developed to a working level.
Next we have the FW-44J (the "J" may not have been given to the test aircraft with the intention of calling it a "jet" in modern terms, as it is a ducted fan with engine heat added) The design reportedly worked but was just abandoned for reasons I can't find. The idea would have eliminated the need for large diameter props with long landing gear and timing gears for machine guns. Plus the design would have opened new aircraft designs to possibility. With the fan being embedded into the airframe.
Next major motorjet is the Caproni-Campini N.1 Motorjet. The Wikipedia page is full of errors and disorganized information. There was two prototypes built, but both had exactly the same name. The aircraft is cited by seemingly everyone for failing to produce performance on par with then current propeller driven fighters. However, since it is known that there was two N.1s built. And from what I can find, it appears that the first one used a 3 stage axial compressor driven by a 700hp piston engine. This aircraft only managed 233mph. Lets call this the N.1a for simplicity. The N.1a and N.1b were 9,200lb aircraft with a nearly 52 foot wingspan. 700hp, even with the rudimentary afterburner, was just not enough power. The other (I'm assuming second) aircraft (we will call it N.1b) but had a three stage axial compressor and a 900hp engine. It also had a redesigned secondary combustion (afterburner). This produced a maximum speed of 320mph. Still only par for the fighters at the time. But it was a comparatively large aircraft for 900hp.
Especially in 1940 when fighters like the BF-109E had a max take off weight of 5875lbs, a wingspan of just 32.5ft and had 1175hp and had a top speed of 348mph.
The N.1a and/or N.1b might have also recorded the 233mph speed at "sea level" and the 320mph speed at a more efficient altitude of around 15,000ft. However, I don't think this is the case as the speed disparity is too high. The other possiblility is that 233mph was without afterburning and 320mph was with afterburning. This seems a more likely case, but doesn't explain why the two models have a difference of power ratings. Unless both aircraft used the same 900hp engine and the 700hp listing is a mistake.
Given the last possibility, where both aircraft were in-fact, identical, then the speed ratings must be with and without afterburning. If the engine ratings are correct, where one aircraft had 700hp and the other had 900hp, then the ratings seem to be based on power differences between the two. That said, it is evident that the thrust from the afterburner DID provide extra thrust, as the aircraft was too heavy and too big to manage such high speed with only 900hp.
Another aircraft with comparable performance was the Macchi C.200 with 34ft-9in wingspan and weighing 4850lbs, the 870hp engine gave the aircraft a top speed of 313mph. Compared to the N.1b with 900hp pushing a 9200lb aircraft with a huge 52ft wingspan to 320mph. It suddenly isn't such a failure. But also, the powers that be probably didn't see the potential and only saw the "same" performance.
Had the N.1 series included another craft with a more fitting 1500hp engine and further improved afterburner, the big N.1 could have been pushed into the 380-410mph range. Which in 1940 still wasn't very impressive, since the P-47 and P-38 already had those speeds. However, where the motorjet would begin to shine is when that 1500hp engine and better afterburner get shoved into a smaller airframe. Also, the compressor design was not as efficient/effective as it could have been with more development than the original that had only 6 blades per row.
The Germans worked on motorjets as well but the war ended before any significant milestones were made.
Heinkel developed engines called the HeS 50z and the HeS 50d. Little is known about them and almost no info exists in the west online. I was able to find some information about the HeS 50z on a Russian website that stated that the engine was a X pattern 4 bank design with 4-cylinders per bank. The engine was almost completely embedded into the engine cowling, with just the valve covers exposed outside. The engine drove a 3-stage compressor and used some of the compressed air flowing around it to feed a small supercharger that fed the engine's fuel injection. The engine was air-cooled, allowing the heat to be added to the flow to increase thrust, and the exhaust was also added to the flow stream. From what I found the 50z had 1200hp. Unknown thrust rating.
The HeS 50d had a horizontal H piston layout with 4 banks of 6 pistons. Reportedly designed to 1500hp, it was liquid cooled and had a radiator in the airflow stream to add heat to the compressed air and cool the engine. The entire engine was within the air stream; probably in an attempt to recover as much heat as possible. The heat normally wasted by an engine can be recovered by a motorjet and is then added to the efficiency, further boosting overall efficiency.
Last in the HeS motorjet series was the HeS 60. The engine was reported to be powered by a 2000hp, air cooled, 32 cylinder radial and produced 2,753lbf thrust with afterburning. The intent was to use them for the Amerika Bomber since they would be more reliable than the jet engines then available. And while the engine was slightly less efficient than a comparable piston engine when running "cold" (without afterburning), it was far more efficient than jet engines at cruise speed. It also was slightly more efficient than turbojet engines of the same thrust at full power. Also, since the engine produced about 40% of max thrust when cruising "cold", it had lots of take-off power but ran efficiently at a slower cruise speed.
In the 1990s NASA worked on a motorjet that was designed to operate at 90,000ft. They wanted the efficiency of the piston engine but needed a compression-thrust system to operate that high.
The MiG-13 used a hybrid motorjet similar to the design of the Napier Nomad. With the aircraft capable of 440mph with propeller alone and 508mph with the motorjet engaged.
..........
Moving back to the what-if.
I think that piston aircraft would continue to push the boundary of 550mph using increasingly ludicrous amounts of power. And while top speed would see a significant diminishing return, rate-of-climb would continue to improve as massive amounts of power became available.
The P-47D had a RoC of 2050-2850fpm without water injection and up to 3260fpm with water injection. The XP-72 had a RoC of up to 5280fpm.
As fighters got slightly larger to accommodate the larger more powerful engines, the need to get traction would become the primary issue. The design of propellers would be the hot ticket area of science. With curved blades, more blades and wider blades eventually growing from the research. Eventually the prop-disc is no longer able to be seen through and the propeller cannot be mounted in front of the pilot. Single engine fighters would need to switch to pushers or twins to maintain visibility.
Meanwhile you would have very small fighters with rocket boosters and/or motorjets trying to make 1250hp engine equipped aircraft faster. The gap between heavy fighters and slight fighters would increase. With light aircraft being used for long range escorts and close air support. While the heavy fighters are focused on interception using their powerful engines to climb up to meet bombers with ever increasing altitudes and speeds. Eventually bombers will be able to fly at 55,000ft at speeds of nearly 500mph. Catching them relies on heavy interceptors with 5,000hp engines with 12, curved, wide-bladed propellers in pusher configuration with rockets on pods that are dropped right after takeoff and initial climb and internal rockets that allow for maximum climb performance upwards of 10,000-15,000fpm. The heavy interceptors level off near the bombers at the same extreme altitude and pursue at 550mph, the tips of the propellers scraping at 0.99 mach.
Eventually ducted fans are used to improve the limits of performance but they still can't break about 550-580mph. Motorjets and rockets are the only way to move forward. 5,000hp air-cooled engines stuffed into big fuselages with 4 or 5 stage compressors and a powerful afterburner. Possibly with small rocket motors inside the afterburner to further improve thrust expansion would finally give level speeds over mach 1. Very small, very thin, swept wings mean very high take off and landing speeds. The gap between fighters and interceptors continues to increase with fighters being nimble and small and interceptors getting quite big to house ever more powerful engines to drive the compressors in ever increasing efforts to catch the bombers. Rocket boosted afterburners and eventually ramjet style afterburners allow for further increased speed. But it would seem that the single engine would reach its limit at 7500hp driving a ducted fan style axial flow compressor. Probably making almost 10,000lbf thrust. Next the interceptors would need two of them in order to keep up with bombers that are already using probably 6 to 10 of these powerful engines to fly up to 70,000ft at speeds of 700+mph. Aircraft would use either multiple motorjets or multiple engines driving a single motorjet compressor.
Rocket powered interceptors would eventually be the only vehicles fast enough to catch the speedy, high-altitude bombers and spy-planes. Using smaller motorjets purely to save fuel on their route and using rockets to zoom to the enemy. The only other method of defense from them would be dirigibles with long range guns and radar hovering at 75,000-80,000ft and forming a defensive line of firepower shooting down at the approaching bombers and spy-planes. Spy-planes soon use rockets once within range to push them to mach 3 for a few minutes. While rocket interceptors are launched from the dirigibles to attack from airborne runways. Aircraft would rely on lots of wide, curved blades, usually in ducts (ducted fans) to get every ounce of tractive force from their every more powerful piston engines. And since a ducted fan creates a jet of thrust, that jet can be augmented by adding heat; afterburners, ramjet augmenters, rocket boosters and other creative methods of making more power would become a race of technical performance. Something we did with turbojets in less than a decade would probably take two decades to even get close. Diesel engines would supplant gasoline when the boost levels got too high for gasoline to cope. Alcohol and diesel would become the more popular fuels. With diesel engines likely boosted well past 100psi and making close to 3hp per cubic inch per pound. Piston engines with close to 10,000hp would emerge being only 60 inches wide but likely more than 200 inches long to make room for so much engine and turbo equipment. The Allison V1710 made 1200hp back in the day, but today tractor-pullers manage to get 3,000-3,600hp from those same engines just with improved fuel injection and modern turbocharging. Imagine a P-51 with 3000hp & an 8 blade prop. Performance would be awesome. But range would not. If the Allison can be made to make so much power, imagine what would happen with other engines. Like the Lycoming XR-7755. It made 5000hp then, today it could probably be pushed to nearly 12,500-15,000hp.
Going by the math that the HeS 60 was 2000hp and made 2750lbf thrust, that means it could manage 17,180lbf - 20,625lbf thrust. But unline turbojets that make that kind of power, it would be a much larger, heavier engine making the power. And if not a motorjet, or even a ducted fan, you would have many-blade, counter-rotating props or rather prop-fans with wide, swept blades. Non-motorjet and non-rocket aircraft would use either ducted fan type cold-motorjets or unducted fans a.k.a: prop-fans.
I know I have rambled on for a while, but it seems that unless you take away rockets and afterburners and even ducted fans, you still end up with a form of jet engine eventually emerging. If you force the world to only use propellers, then you end up with unducted fans with many huge swept blades and almost exclusively counter rotating in order to capture that immense power.