Oracledatabase12.2. It has nearly 25 million lines of C code.
How terrible this is, it is hard to imagine! You can't change a single line of code in a product without breaking thousands of existing tests. Generations of programmers have written this code for a limited project period, which is full of junk code.
Very complex logic, memory management, context switching, etc., all connected by thousands of flags. The entire code is full of mysterious macro commands. If you don't take out your notebook and manually expand the relevant macro commands, you won't be able to understand these commands. It may take a day or two to really understand the role of a macro command.
Sometimes you need to straighten out the values and effects of 20 different flags to predict how the code behaves in different situations. Sometimes as many as hundreds of flags! This is no exaggeration.
The only reason this product is still alive and still available is millions of tests!
The following is the Oracle databaseDevelopmentDaily routine of personnel:
Start working on a new bug.
It took two weeks to try to understand 20 different flags that interacted with each other in a mysterious way, leading to this dilemma.
Add another flag to handle the new special scene. Add a few lines of code to check this flag and fix the problematic situation to circumvent the bug.
Submit changes to include approximately 100-200 unitsserverTest server clusters that compile code, build new Oracle databases, and run millions of tests in a distributed fashion.
Come back home. Come to work the next day and continue to deal with other bugs. The test may take 20-30 hours to complete.
Go home again. Come back to work and check your cluster test results. If it goes well, there will be about 100 failed tests. If you are unlucky, there will be about 1000 failed tests. Randomly select some tests and try to figure out what went wrong with your assumptions. Maybe you need to consider more than 10 flags to really understand the nature of the bug.
Add some flags to try to solve the problem. Submit your changes again to test. Wait 20-30 hours.
Repeat for two weeks, until you get the "mystery spell" that combines these flags.
One day, you will succeed and no longer fail the test.
Add more than 100 tests to your new changes to ensure that the next developer who unfortunately touches this new code will never break your fix.
Submit the results of the final round of testing. Then submit for review. The review itself may take between 2 weeks and 2 months. So go ahead and deal with the next bug.
After 2 weeks to 2 months, everything is ready and the code will eventually merge into the main branch.
The above is a description of the daily life of programmers who fix bugs in Oracle, and it's no exaggeration. Now imagine how terrifying it would be to develop new features. It takes 6 months to 1 year to develop a small feature (it may take 2 years to add a new authentication mode, such as support for AD authentication).
This product is a miracle in itself!
I no longer work for Oracle. Never work for Oracle again!