Two days and a countless …(censored)… later, I think I figured it out. Here’s the mystery:
- I successfully compile my Java code into a bunch of Java archives (JARs).
- I deploy that code to my server.
- The app fails to start. The stack trace shows:
- some of my code runs
- my code invokes some of my other code, but instead of getting there
- an ExceptionInInitializer is thrown, although the class might not even have an initializer
- caused by a (raw, not a subclass) RuntimeException, with an error message that starts with “Uncompilable source”; the rest of the message varies.
- note that in spite of the error message this is at run-time, not at compile time (compiling went fine). It does not involve JSP or other late-compiled code.
- Sometimes, a squeaky-clean recompile / re-install fixes the issue. Sometimes it does not.
Needless to say, my code does not throw a RuntimeException with a message of “Uncompilable source”. But neither does the JDK (I grepped the sources, repeatedly), nor Tomcat, nor any other Java library I’m using.
The debugger has nothing to say on the subject either, for it, the code that throws the RuntimeException does not exist.
What would you do? (Spoiler alert aka the solution following.)
So in a major act of desparation on day three, I unpack javap, the Java bytecode decompiler. I decompile some of the files that might be involved using
javap -s some/package/myclass.class
and after some poking around, I think I need more coffee:
Some file that I just had decompiled, and then decompiled again, looked different. It appears much longer all of a sudden. How can that be? Am I dreaming? Confusion descends on your already-confused correspondent. And while I’m being confused, all of a sudden, a “Uncompilable source” text string scrolls by my wary terminal when decompiling a file that did not have it in it just a minute ago. What?
Here I need to say that I’m using NetBeans as my IDE. I don’t use it to compile (that’s outside, in maven, which is a piece of work in itself, but I digress) or run (that’s in a different Linux container). I do sometimes use its debugger to connect to my Linux container over the network.
It turns out NetBeans, running in the background, rewrites the .class files while I attempt to browse through them. Never mind that I just cleanly compiled them and that there’s absolutely no need for that. It rewrites them, and guess what, inserts some byte code whose comment says it will throw “Uncompilable source”. When I check in NetBeans, it turns out it must be confused about the state of its cache or something else that nobody can understand, claims that some symbols do not resolve (they just did), and overwrites perfectly-well compiled .class files with bytecode that, when run, throws that unbelievable exception.
So this is what happened:
- I compile my code just fine.
- While the (sometimes lengthy) compilation is ongoing, NetBeans, which I paid no attention to because it wasn’t involved, immediately overwrote some of the class files with the equivalent of “but I don’t want you to run this, and I won’t take responsibility for it either” code.
- Which the build process then packaged up into the JARs, instead of the correct code.
Which genius wrote this code? I’d like to send you a bill for 2+ days of my time.