Subscribe: Tom Hawtin
http://jroller.com/rss/tackline
Added By: Feedage Forager Feedage Grade B rated
Language: English
Tags:
access  classes  code  don  java  language  lock  make  method  methods  new  object  public  security  static  thread 
Rate this Feed
Rate this feedRate this feedRate this feedRate this feedRate this feed
Rate this feed 1 starRate this feed 2 starRate this feed 3 starRate this feed 4 starRate this feed 5 star

Comments (0)

Feed Details and Statistics Feed Statistics
Preview: Tom Hawtin

Tom Hawtin



Thomas Hawtin's Weblog



Last Build Date: Mon, 13 Aug 2012 01:21:38 -0400

Copyright: Copyright 2012
 



The Next Big Language in the Java Line

Tue, 16 Nov 2010 11:22:50 -0500

What do we want?

A substantial productivity boost, fewer bugs, security we can trust, more responsive to change and better documentation.

When do we want it?

Ooh, in about 10 years.

C++ brought in major changes over C (though allowed some kind of mixing the two languages). Compare string handling code in Modern C++ and C. Java comes in with garbage collection, robust typing, better definition, less mess and generally a language that feels trustworthy.

The next big language in the Java line will not be a JVM language.

Over the years, Java, Java dialects and Java-like languages (a language is a dialect with sales and marketing) have been successful on non-JVM VMs:

  • Javacard VM
  • CLR
  • Dalvik

Java the language is the common factor. If you can change the VM without changing the language, why would you keep the VM when you replace the language?

Languages which content themselves to feaping creaturism and fiddling with minor syntax are not going to cut it. Arguing over which colour to paint the deckchair whilst on fire is confused. Alternative JVM languages may gain popularity if Java becomes bloated with features, leaving less reason to stick with it.

The next big language will need similarly big changes that C++ and Java brought. Much of the low hanging fruit has gone.

  • Syntax. This has been neglected. Possibly the biggest problem with Java code is just reading it (though fortunately it's relatively easy to understand once you've got past the noisy syntax). It's easy to fix. Fix it.
  • Transactional Memory. What really convinced me about this is the reaction to this has been reminiscent of garbage collection. Effective operation will be helped by other features, but efficient implementation will make GC look like a piece of piss.
  • Abstraction of threading. A thread should not be aware of its thread. Not only does this mean no `Thread.currentThread`, but it also requires reformation of exceptions (and locking, but we have transactions).
  • Encourage immutability. As well as helping STM, it's very helpful from a programming perspective. C++ has implicit object copying all over the place. Java uses references uniformly, which should mean that common types, such as `List<>`, should be immutable.
  • No nulls. Even before generics, NPEs were vastly more common than "CCE"s. A null contains little meaning. Let's get rid of the menace.
  • Recognition that value and reference types are different beasts.
  • Arbitrary sized integers. I know what you're thinking, but with an appropriate compiler this should not be a significant performance concern.

Note, this is not a template for a language. I would like a language with: a preference for composition, declaration-side variance, more explicitly typed, stronger typing, "smart" target typing, better modularisation than just package & class, better construction support, etc., but no multiple inheritance of any kind, access modifiers on members or bitwise operators. The above points are what I feel to be pretty much the necessities.

C and C++ are now only fit for legacy and niche applications (and programmers stuck in the past). Migration assistance is a trick Java missed in the craze for doing everything over again in internet time. In time, Java will become legacy only.

(Seeya at Devoxx this week.)




sbb.ch: Swiss inefficiency

Thu, 5 Jun 2008 16:44:10 -0400

Trying to book tickets to get me to and from Jazoon08. Shouldn't be too difficult. I'm used to the dropkickery of UK rail sites, but the Swiss are supposed to know how to do things.

Among the many fuckups:

  • Insists on using the current time of day, no matter what is specified. Here's a clue: avoid JavaScript whereever possible, for it is shite.
  • Only shows a few trains, forcing pointless page round-trips to show more.
  • Wont show prices without adding going through pointless forms for each bloody train and class.
  • Loads of pointless forms.
  • The usual cockups from inappropriate state in the session.
  • Requires seat preferences whether or not it is going to reserve a seat or not.
  • Times out tickets after twenty minutes. I needed two tickets. It took over twenty to sort through the nonsense (even when I knew what I was looking for).
  • Wont take my frikkin Laser/Maestro.

Jim Coplien has stated, roughly, that bad usability (particularly websites) causes many deaths through causing unnecessary stress. I agree.




New weblog

Sun, 25 May 2008 11:55:15 -0400

I have a new weblog on my employer's blog site, in which I shall completely fail to go into details about what I am working on. I'm sure you can guess the URL.

I will add to this blog occasionally, but it will be the concentrated residues of my unprofessionality. You have been warned.




ACCU 2008 Oxford Conference, Rambling comments on

Sat, 3 May 2008 12:27:34 -0400

[I wrote most of this about a month ago. If I don't post it now, I never will do. I'm sure there were a couple things I meant to add.] It's been years since I last went to an ACCU conference. Back then it used to be held in the middle of town. Now it's in a poncey hotel on an out of town roundabout. I stayed in a very nice guest house at the other end of Five Mile Drive (which isn't actually five miles long and doesn't have cattle). Being a hotel conference centre the screens are low, leading to ninety minute sessions of looking at the back of someone's head. And having someone looking at the back of your head. From a show of hands it seems as if around 80% of the attendees were C++ programmers. My main interest in C++ is not in the language (other than as a safer C), but in the wild and ingenious solutions to problems that shouldn't exist. I didn't really get Irish jokes when I was a kid, I thought the Irish was ingenious and mischievious. I wouldn't want to live in C++. In one break I came out of the smallest room to find a bloke in front of me loudly claim that all the idiots moved from C++ to Java. Indeed in the "State Of The Practice" panel at the end of the conference it seemed widely agreed (by the audience at least) that if you could write in C++ you had to be smart. Chronic poor judgment, but smart. I suppose to the man driving the Clapham omnibus anyone who can get a shoddy piece of C++ to compile is smart. In his keynote, Andrea Alexandrescu talked about how the C preprocessor was complicated enough to have experts in ## and C++ has experts in functions that do nothing. Can we at least try to keep the language relatively easy to understand. My expectations of Roger Orr's keynote on debugging weren't that great. I mean debugging, it's not very sexy. But actually it turned out to be one of the best and most entertaining sessions of the four days. A bit too much in the show of hands department, but other than that surprisingly lively and interesting. Aladair Meredith (who once interviewed me - I didn't get the job) did 90 minutes on C++ 2009, for sufficiently large values of 90 and 2009. C++ 97 was hardly Ally McBeal to begin with. So lots and lots of little features. Pretty much all of them individually good. But you've got to think there's a better way. Andrei Alexandrescu did a talk on his D 2.0 language. This is a language starting from C++, then going safe. So garbage collection, no dynamically unsafe casts, array bounds checking, etc. I was amazed how far he managed before breaking down and making a comparison to Java. D is closer to C++ than Java is, but the similarity is clear. The transitive const is interesting. It concerned me that C++ const doesn't do quite as much as you'd expect. But going transitive (and with no accessing external variables) seems a little too constrainted. How about memoization? The approach to construction of transitively invariant objects also seemed a bit of a hack, but it is a tricky problem. Now I haven't seen that language so I don't know if it can handle these sorts of things. Arrays are handled with "fat pointers" that are very much like the way (JDK/non-Azul) Java implements Strings. The pointer includes a pointer to the base of the array, an offset and a length (or equivlent set of values). Seems a bit inefficient to have the overhead of slice support everywhere. It seems the idea is to use an array directly. For instance a D 2.0 string as an transitively const character array. Presumably external methods may be grafted on. But still, the way mutable and immutable values are treated and operated upon are fundamentally different. Also an array only gets you so far. If you want a list or mutable string that you can append to, mutable arrays are going to have to be wrapped, which means duplication. I went to two workshop sessions. The simplicity workout was a bit crowded. Suits me as I c[...]



User definable synchronous control structures without closures

Fri, 1 Feb 2008 06:55:13 -0500

User definable control structures do not require closures. This is not to say that encouraging dialects is necessarily desirable (I don't think it is), but merely possible and possibly better without closures. Closures introduce some surprising semantics which we could do without. This unproposal is just for control structures - threads and listeners are right out. People of a web persuasion may find this somewhat familiar. Here follows an example of one way to achieve control structures without closures. We want to end up with something that will allow us to, say, loop through key-value pairs of a map as: for keyValue (MyKey key, MyValue value : map) {     ... } The body isn't a closure, it's just plain normal code that is part of the method. So, start with a type KeyValue that represents a particular control structure. It could implement a standard interface but there is really no necessity. The type only needs a method (or overloaded methods) that returns an instance of a control structure. This is much like Iterable.iterator. We'll call these methods forOf, tryOf, etc., depending upon which keyword is used to introduce the structure. The parameters of the method are populated by the arguments right of the colon in the for statement above. public interface KeyValue {     Controller forOf(Map map);     Controller forOf(Iterable> entries);     interface Controller {         ...     } } We also want a instance of this type available. public final class Structures {     public static final KeyValue keyValue = KeyValueImpl.instance;     private Structures() { throw new Error(); } } So onto the control structure instance itself. What do we need? We already have the arguments right of the colon. We need something to determine whether the body should be executed (again), equivalents of catch/finally and someway of getting the loop variables. We can define an enum to determine what the code should do next (actually a set of ints is more likely for low-level performance reasons). package java.lang.control; public enum Action { EVAL_BODY_AGAIN, EXIT } It's probably worth defining a separate method for testing before the first iteration and subsequent iterations. @Override public Action doStart() {     if (iter.hasNext()) {         entry = iter.next();         return EVAL_BODY_AGAIN;     } else {         return EXIT;     } } @Override public Action doAfterBody() {     return doStart(); } Structures such as using need not define the latter method. In fact using would want to define that doStart always evaluates the body (or throws) - perhaps by having a void return type. Catch and finally are straightforward. Again because the compiler rather than a library is doing the work, we can overload catches without doing something odd to generics. Finally can be broken into two cases: exit due to break/continue and an uncaught exception. For instance: class ControllerImpl implements Controller {     ...     @Override public Action doCatch(         SQLException exc     ) throws MyException { ... }     @Override public Action doCatch(         IOException exc     ) throws OtherException { ... }     @Override public Action doCatch(         EXC exc     ) throws EXC, ResourceException { ... }     @Override public Action doBreak(     ) throws ResourceException { ... } } Arguments can be read with a series of get methods:     @Override public K get0() { return entry.getKey(); }     @Override public V get1() { return entry.getValue(); } So in the example above, javac would cr[...]



An alternative to flexible checking for rethrown exceptions.

Sun, 13 Jan 2008 17:15:31 -0500

It has been suggested that the Java language should be extended to allow caught checked exceptions to be rethrown without explicitly mentioning the exception type.

void fn() throws XException, YException, ZException {
    try {
        ... throw [XYZ]Exception ...
    } catch (final Exception exc) {
        ...
        throw exc; // Must be of type caught.
    }
}

A more Javaish alternative would be to make the catch clause generic, much like a method.

    } catch (EXC exc) {
        ...
        throw exc; // Must be of type caught.
    }

The default bound could be to extend Throwable, rather than the usual Object. So for any type of exception:

    } catch (EXC exc) {

We need to be slightly more restricted than with methods, as we can't catch, say, Exception & Iterable.

Just a thought.




checkPermission: SecurityManager vs AccessController

Thu, 10 Jan 2008 16:11:55 -0500

So does AccessController.checkPermission completely replace SecurityManager.checkPermission, as was suggested in a recent Javalobby mailing? No. (Despite the poor example in the AccessController API docs.)

Pity, because as a static method, the AccessController version is simpler to use.

The first problem is that you don't want to call checkPermission if there is no current SecurityManager, so you still need to System.getSecurityManager() . More fundamentally, SecurityManager instances can contain functionality rather than just changing the interface. Now, this isn't normally done by overriding the checkPermission methods, but the abstract notion of principle remains.

For example, suppose you wanted to restrict/derestrict access to a Thread (in the sense of RuntimePermission "modifyThread"). You could do this by overriding SecurityManager.checkAccess(Thread). Perhaps you might want to take into account the current thread. These sort of things are not possible through AccessController and Policy.

For permissions that are not covered by the special purpose checkXxx methods, this sort of code needs to go into overrides of SecurityManager.checkPermission. Don't go over this method's head into AccessController.

So where can one use AccessController.checkPermission? Implementing java.lang.SecurityManager for one. It could also be used to check that you are not elevating privileges. Pretty obscure stuff.

Just don't ask about SecurityManager.getSecurityContext.




Builder language support

Tue, 25 Dec 2007 12:38:00 -0500

Builders are a good thing, but the code tends to look a bit messy and verbose in a bad way.


    Frame.Builder builder = new Frame.Builder();
    builder.title("My Frame");
    builder.bounds(0, 0, 400, 300);
    builder. ... ;
    builder. ... ;
    builder. ... ;
    Frame frame = builder.create();

It appears that most Java programmers that care about such things use the familiar returned this hack.


    Frame frame = new Frame.Builder()
        .title("My Frame")
        .bounds(0, 0, 400, 300)
        . ...
        . ...
        . ...
   .create();

There's even an awkward proposal to make the language support implicit returning of this.

To me the concept of the proposal seems to confuse two things. On the one hand we have the client code, and the other the implementation. The implementation isn't that interesting or difficult. The language feature does nothing to help client code. Its biggest significance is allowing chained calls on APIs that were not designed for it.

I suggest starting from a different hack.


   Frame frame = new Frame.Builder() {{
        title("My Frame");
        bounds(0, 0, 400, 300);
        ... ;
        ... ;
        ... ;
    }}.create();

(If you've not seen it before and don't get it, it's an anonymous inner class containing only an instance initialiser.)

The non-bullshit problem with that is that adding classes adds to runtime size. You might be using anonymous inner classes all over the place, but this looks wasteful1. If you had lots of little anonymous inner classes implementing the same type, you might as well compile to a single class file with a switch.

So a general solution would be to add a tiny bit of syntax to allow the compiler to share implementations between anonymous inner classes, or just not extend at all. Say squeeze a dot between ) and {.

(Also remove final from StringBuilder.)

Adding this syntax would reduce the gunk in code, and also reduces the permanent generation.

1 Even if 1 K at the time of writing is worth the about same as 1 byte when I started university. Well, perhaps you might expect more from a computer these days. Pity they can't reach the usability of my arc.

Comments on comments

jroller is eating my own comments, so I'll answer here. I have also rescued some comments marked as spam.

Rob: So C# has invented an extra syntax, and can't manage anything other than a sequence of assignments? Sometimes it's useful to limit flexibility, but I'd rather not. IIRC, GoF uses a maze for its builder example, that wouldn't work with the C# syntax.

Remi: JavaFX does some nice things, but I don't want a whole new language for some minor "syntactical sugar".

Stephen: Given my employer, KSL makes more sense...

Macneil: Creating builders with factories seems on the face of it a bit of an odd thing to do. OTOH, for generic builders that would make up for the lack of inference on new.

Bruno: I think that would confuse too much.




Static this.getClass()

Thu, 22 Nov 2007 14:52:59 -0500

So there you are in a static context and you want to get the Class of the enclosing class, but you don't want to have to name it explicitly. What is one to do?

    new Object() {}.getClass().getEnclosingClass()

(from 1.5)




Method-local methods for recursion

Fri, 4 May 2007 00:33:13 -0400

Often when writing recursive algorithms, the recursive method doesn't have a suitable interface to be published. The interesting method should be hidden, with a second, public method introduced to check arguments, set things up, do a bit of munging and tidy the furniture. However, conceptually there should be just one method. I generally start off writing a single method and then split (perhaps that makes me look like a bit of a recursion n00b).

A "proper" language would have support for recursion at this sort of level. How can we in Java express something that at least looks like local methods? Local classes are the obvious, if obscure, solution. But who wants to fill the place up with all that clutter?

It occurred to me whilst in the bath, that we can use a little known property of anonymous inner classes to avoid fiddling with named types. Although anonymous inner classes have no name, they do define a type. I was thinking that you can only use this once - immediately after the new expression (you can pass the value through capturing generic method calls, but that doesn't help much). However, the type is, of course, also implicitly available within its own methods, which is ideal for recursion.

So, using the well-worn quicksort example (I never liked sorts) and some stolen code:

public static void sort(final long[] array) {
    if (array.length <= 1) return;
    new Object() {
        private void sort(int off, int len) {
            ... partition ...

            // Recursively sort non-partition-elements
            if ((s = b-a) > 1)
                sort(off, s);
            if ((s = d-c) > 1)
                sort(n-s, s);
        }
        private void swap(int a, int b) {
            long aOrig = array[a];
            array[a] = array[b];
            array[b] = aOrig;
        }
    }.sort(0, array.length);
}

If you don't do much recursion, you'll be better off sticking to the old methods. Particularly if you don't want your colleagues to throw things at you. However, with a lot of recursive code it might be worth considering.




My take on closures

Sun, 18 Mar 2007 04:11:52 -0400

"Everyone" is having their say on what should be done for "closures" in JDK7. Here's my take. Write access to local variable of enclosing classes. Types (or just individual generic arguments) for constructors/anonymous inner classes can be omitted, using defaults class specified by type. Braces not required for anonymous inner classes that only (explicitly) define one method. Local class methods that override/implement default to public access. package member modifier acts as package private. Inference of return types and throws clause for anonymous class non-abstract methods without explicit modifiers. Inference of method name for anonymous class non-abstract methods with inferred return type. Short syntax for methods that just return an expression. private suffix for disposable expressions. Write access to local variable of enclosing classes. I really hope this isn't controversial. I don't think there is any need to mark the variables in any special way. Types (or just individual generic arguments) for constructors/anonymous inner classes can be omitted, using defaults class specified by type. We know if we want to create a new List we want a new ArrayList. If we want pass an anonymous inner class to EventQueue.invokeLater, there is no way we are not going to implement a Runnable. So new syntax.     List strs = new ArrayList();     List strs = new();     Map strs = new WeakHashMap();     Map strs = new WeakHashMap<,>();     EventQueue.invokeLater(new Runnable() { public void run() {                 frame.setVisible(visible);     }});     EventQueue.invokeLater(new() { public void run() {                 frame.setVisible(visible);     }}); The supertype declares the default intermediate type. Modifying default is an incompatible change, but adding one is not.     public interface List     extends Collection     default new ArrayList     {     public interface Runnable     default new Runnable     { Braces not required for anonymous inner classes that only (explicitly) define one method. It's noise we can do without.     EventQueue.invokeLater(new() public void run() {             frame.setVisible(visible);     }); Local class methods that override/implement default to public access. There is no language level access to them that could be denied. There is nothing to be gained by explicitly stating the access modifier.     EventQueue.invokeLater(new() void run() {             frame.setVisible(visible);     }); package modifier acts as package private. It's an anomaly that the most obscure access level is the default. For pedants this allows keeping package-private access prevented by the previous point. Inference of return types and throws clause for anonymous class non-abstract methods without explicit modifiers. We are not using the type. The type is being used by algorithm method. We wouldn't explicitly state the type of an actual argument when invoking a downcall. This fits in with callback methods not checking their arguments, but their return values should be checked by the calling method. The return type in this situation is just noise.     EventQueue.invokeLater(new() run() {             frame.setVisible(visible);     });     statement.query(new RowHandler() { public void row(ResultSet[...]



TypesDefinePieces

Tue, 9 Jan 2007 16:00:45 -0500

Neal Gafter has a weblog entry on how Smalltalk MethodNamesInPieces could be used to allow closures to be used in expression of complicated flow control. (Read that first.) The obvious inner class approach is to add method for each block. However, this does not scale too well. How, for instance, are we going to remember what order the methods should be in? If they are in an odd order, there could be surprises. We can do something with the Lee Lea Bloch blocks (assuming type arguments are inferred).     public String toString() {         final StringBuilder sb = new StringBuilder("[");         forEach(             someCollection,             Each(String s) {                 sb.append(s);             },             Between() {                 sb.append(", ");             }         );         return sb.append("]").toString();     } Where we have a method statically imported:     public static void forEach(         Iterable elements, Each each, Between between     ) Now, this could be extended with First, BeforePenultimate, etc. But how many times would we have to overload forEach? A slightly different approach is to introduce a base type (say an abstract class with a private/package private constructor). and use run-time type checking.     public static void forEach(         Iterable iterable, ForEachSection... section     ); (Okay, so there is a slight problem with that last parameter declaration, at the moment.) The use of types to indicate rolls feels a little like Data Attribute Notation, but not too much. Translating back to Gilad, Gafter, Gosling, Ahé closures, the code becomes slightly messier (although the public interface is more compact).     public String toString() {         final StringBuilder sb = new StringBuilder("[");         forEach(             someCollection,             each({String s =>                     sb.append(s);             }),             between({ =>                     sb.append(", ");             })         );         return sb.append("]").toString();     } [Update 10-Jan-07: It appears that I originally used stale syntax. However, it appears we are stuck with parentheses as well as curlies because BAGG treats commands and expressions differently. This, in my opinion, runs against the spirit of the Principle of Abstraction (Principles of Programming Languages, R.D. Tennent, p114). Original code is in the page source.] Where each and between methods are statically imported. (I think that's correct. In this case the blocks contain single expression statements, so I could have used colons instead of braces and semicolons. However, I don't think that helps legibility any.) It's all a bit of a silly example. I'd write it thusly.     public String toString() {         final StringBuilder buff = new StringBuilder();         String sep = "[";         for (final String str : someCollection) {         &nb[...]



"Secrets Of The Masters" (allegedly)

Fri, 29 Dec 2006 13:28:34 -0500

Linux Sys-Con hava a new (Dec 2006) article listing 30 sample interview questions for Java "masters". So let's have a look at some of the more advanced ones...

For senior-level developers:

Yes, "senior-level" developers.

** Q21. What's the difference between a queue and a stack?

A. Stacks works by last-in-first-out rule (LIFO), while queues use the FIFO rule

To be a pedant: If we are talking Java, java.util.Queue (capital Q) can be LIFO too. In fact it would take much to get Stack to implement Queue (it may, however, surprise people - don't be surprised).

...

** Q27. How would you make a copy of an entire Java object with its state?

A. Have this class implement Cloneable interface and call its method clone().

No! Do not implement Cloneable. It is pure evil.

** Q28. How can you minimize the need of garbage collection and make the memory use more effective?

A. Use object pooling and weak object references.

Which is also an exceeding good way of increasing garbage collection times and wasting memory.

** Q29. There are two classes: A and B. The class B need to inform a class A when some important event has happened. What Java technique would you use to implement it?

A. If these classes are threads I'd consider notify() or notifyAll(). For regular classes you can use the Observer interface.

No! Do not implement java.util.Observer! If you want to apply the observer pattern, I strongly suggest using the Java Beans event conventions (well perhaps not going as far as implementing the getXXXListeners nonsense).

Is it significant that A and B are classes? If it is in a threaded context, consider java.util.concurrent first.

** Q30. What access level do you need to specify in the class declaration to ensure that only classes from the same directory can access it?

A. You do not need to specify any access level, and Java will use a default package access level.

Shouldn't 'directory' read 'package'? This is the most advanced question needed for a technical interview?

Verdict: Above industry standard.




Intercepting events without broken glass.

Thu, 21 Dec 2006 17:36:58 -0500

The usual way of intercepting mouse events is with a glass pane. Other than being a bit clumsy and intrusive, there are many problems with glass panes to do with PL&F interference, D&D, etc. An alternative is to intercept the events as they leave the EventQueue. Applets and JNLP applications can push their own event queue. Applications started with -Djava.security.manager cannot. Further, overloading the method we need, dispatchEvent, prevents C&P working with the system clipboard. As dispatchEvent is protected, we cannot operate as a chain of command, and so there can be only one custom EventQueue (at a time, per 'app context'). At the point the event is intercepted, determining the correct lightweight component has not happened. So in order to check whether the event is of interest, we need to recursively check which child the event hits. If Swing is assumed, there is no need to check parents (for the case of heavyweight children). The example code is a bit basic. There are no checks for drop targets or key event handling, for example.     /**     * Keys form weak set of component to intercept mouse events to.     */     private static final java.util.Map intercept =         new java.util.WeakHashMap();     /**     * Install event queue.     * Note, we can only do this once per 'app context'.     */     static {         EventQueue eventQueue =             java.awt.Toolkit.getDefaultToolkit().getSystemEventQueue();         eventQueue.push(new EventQueue() {                 @Override                 public void dispatchEvent(java.awt.AWTEvent awtEvent) {                     if (awtEvent instanceof MouseEvent) {                         MouseEvent event = (MouseEvent)awtEvent;                         Object source = event.getSource();                         if (source instanceof Component) {                             // (If we were interested in heavyweights,                             //    we would also check parents here.)                             if (isIntercept(                                     (Component)source,                                     event.getX(),                                     event.getY()                             )) {                                 // Swallow.                                 return;                             }                       [...]



Review: Java Security Guidelines by Gary McGraw and Ed Elten

Sun, 17 Dec 2006 17:03:05 -0500

This book, Securing Java, dates from 1998/1999, but that's quite modern for Java mobile code security. Section 1 of Chapter 7 "Java Security Guidelines: Developing and Using Java More Securely" presents 12 rules for writing security-critical Java code. These were also published in JavaWorld (but they do not permit linking to a sensible version of the article). Rule 1: Don't Depend on Initialization As far as I am aware, objects are recoverable even without completing constructors normally in the following ways: If java.io.Serializable is implemented but not java.io.Externalizable, then the no-args constructor of the most derived non-serialisable class is used, and readObject methods act as psuedo-constructors. Before JLS third edition clarified matters, objects are recoverable through a finaliser. Sun's 1.5 does not conform to this change. From 1.6, supposedly the Object constructor has to exit before the object is finalisable. However, subclass constructors may not have had super return. Constructors can make checks before calling super, but cannot determine subclasses. Constructors can leak this into passed in arguments, static variables or thread locals. Theoretically, unsafe publication can lead to incompletely constructed objects visible from other threads. Given the rules concerning final fields, I would be very interested to see a practical example of this. Using an is initialised variable should be seen as a last resort, to be used when a poor public interface locks down alternatives. It's an error prone mess. Technically the variable should be volatile to guard against unsafe publication. A better approach is to make sure that the object or class is never in an insecure state. Rule 2: Limit Access to Your Classes, Methods, and Variables Package-private is as good as private. Rule 3: Make Everything Final, Unless There's a Good Reason Not To Non-public/protected is good enough. Adding final to methods routinely adds noise. OTOH, inheritance can allow really nasty stuff. Rule 4: Don't Depend on Package Scope Bullshit. Also: Only the bootstrap class loader can load classes into any package with a name starting with "java.". You can use the security manager to prevent access to other package prefixes. Packages can only be loaded by a single class loader (although namespaces may be shared). Sealed classes would be pointless. Rule 5: Don't Use Inner Classes Bollocks. Rule 6: Avoid Signing Your Code Yay! I'm sure I saw some guidelines suggesting you might as well sign code(!), but I can't find it now. Note, you don't need to be within doPrivileged to be privileged. Rule 7: If You Must Sign Your Code, Put It All in One Archive File Not a bad idea. It appears that mix-and-match attacks are non-trivial in practice. By default jarsigner will sign the finished manifest. This means you can't just pick and choose classes. So long as you do not change that default! For the paranoid, JNLP supports placing a signed copy of the .jnlp file within the archive. This will only be effective for an attacker targeting JNLP. I further suggest putting all your classes in one package, and all the classes non-public. Perhaps use a bit of sed, or equivalent. Et voilà, you have your own super packages. This also applies to your bootstrap classes loaded by the system class loader if you do not restrict access to those packages (the system class loader is available to attackers). Rule 8: Make Your Classes Uncloneable Highly obscure, but valid. You might as well leave the clone method as protected rather than advertising it as public. In some obscure cases it is possible to subclass a sensitive object and have a clone confuse the internal [...]



Review: Sun's Java Code [Security] Guidelines

Sat, 16 Dec 2006 20:20:54 -0500

For many years this longstanding document appears to have represented the Sun's advice on secure mobile code. Below is a review of each individual point. The completeness as a whole, I leave to your imagination. Note at JavaONE May 2006 mention was made of updating the document. This has not yet happened. Static fields Yes, non-final public statics are bad. As are non-final protected statics (I still don't understand what they are supposed to imply). Also final public/protected statics referencing mutable objects. For instance, java.awt.Component is very mutable, as are arrays. Applies even if the static's static type does not support any mutation operations. As second bullet, hiding behind accessors or other types of method is not necessarily effective. Reducing scope As with the last point, a good general guideline. However, tightening from package-private to default does not actually help security other than to make the code more readable (unless you are doing something very odd with reflection...). Public methods and fields Again, include protected. Remember serialisability constitutes a public constructor, even if readObject/writeObject are not present. Protecting packages All package are proof against insertion from other class loaders. Child class loaders can use the same namespace, but the classes do not benefit from package access to the parent loader's classes. I believe if you load a class ahead of a sealed jar, it will get inserted. Classes with the same namespace but different package loaded by a parent class loader may replace completely supposedly secure classes from child class loaders. The equals methods This section is very terse. What can we say about the equals method? User supplied objects may not behave as expected. Make objects immutable if possible Yup (and final). With the new (1998) collections framework you can return an read-only view. However, you still need to make sure that the read-only object does not contain mutable objects. Never return a reference to an internal array that contains sensitive data "Now I know technically speaking that's only one drawback, but I thought it was such a big one, it was worth mentioning twice." Never store user-supplied arrays directly Or generally mutable objects. And mutable objects contained by objects. Remember that methods on the object that purport to copy may not do what is expect if it is subclassable or delegates. Serialization If it's serialisable, it's probably insecure. transient, or equivalent mechanism, good. You should ensure that at no point during deserialisation is any object in an insecure state. ObjectInputValidation has virtually no merit as a security mechanism. The write array methods are bad. writeObject and writeUnshared are viable alternatives to cloning the array (though probably not actually faster). No mention is made of read or readFully. readObject is not an acceptable substitute. Freakishly enough it appears that neither is readUnshared for byte arrays (not checked). Encrypting a byte stream seems a bit out of place here. Is encryption only or even especially useful for serialisation? Implementing Serializable is like adding a public constructor, even if the class is not itself public. So if you were relying on private constructors or package-private classes, you will need an isInitialised flag or similar, and an awful lot of checking. This is v. funny (IMO). Native methods I guess the important thing about native methods is the dropping of all the Java goodness of bounds checking, automatic memory management, etc. Clear sensitive information How helpful this is, I don't know. You j[...]



Security flaws in untrusted code

Fri, 15 Dec 2006 13:01:42 -0500

Surely not possible. At worst sensitive data is sent in the clear. Well, if you try really hard, you can introduce flaws.

The approach presented here works in a shared execution environment with an appropriate security manager, such as the Java Plug-in. What you will need to do is create a thread whilst running on a system thread. The security manager will assign this the applet context thread group, rather than the current thread group. The finalizer thread is the obvious example of a system thread. You will also need to create a thread from that thread. I said you had to try hard.

For exploitation, InheritableThreadLocal is just the ticket. It has the useful properties that it is inherited by threads(!) and that the childValue method is called on it when a new thread is created. childValue is called from the thread doing the constructing, not the constructed thread. Hence the need to create a thread from the thread created on the system thread.

class Genie extends InheritableThreadLocal {
    @Override
    protected Object initialValue() {
        return this; // Leaks in current implementation.
    }
    @Override
    protected Object childValue(Object parentValue) {
        ... try to do evil ...
        return parentValue;
    }
}
        // Kick off evil in system thread.
        // get required to set it off.
        new Genie().get();

Now the attack code is in a foreign applet context. This applet context is determined first by thread group and only if that is inconclusive does the stack get a look in. As the applet is not signed the client machine is safe. How to be evil?

With sufficient cunning, the target applet could be used to launch a denial-of-service attack on its server. Not very interesting. The attacker can use Frame.getFrames or Window.getWindows to get hold of an manipulate private data (say passwords). Perhaps the applet was loaded from an internal web site, which the attacker can now browse. Any authentication that the applet had access to may be accessible.

In conclusion: Don't start threads from finalisers. ;)

See:




Mixed certification: An example

Thu, 14 Dec 2006 18:06:43 -0500

As a change from being all theoretical, here is a practical way to exploit naively signed code. Instead of picking on a kosher signature, I have used an unverified signature (anyway, the colours nicely match the warm and friendly Ubuntu look). The JDIC FileExplorer demo happens to be the first signed applet/application I deliberately looked for security flaws in. Rather conveniently, FileExplorer clears the security manager. If we can get our code to run after this, then the sandbox is down. First we need to set up our JNLP application. First we create the main application .jnlp that references the extension.       Descriptive Title     Who?                             Funnily enough, the title appears on the trust dialog as if it were part of the certificate. We have asked for all-permissions, which is okay because the only jar directly referenced is signed. Next we need the (unsigned) extension .jnlp:       Malicious Extension     Someone untrustworthy             fileexplorer.jar is downloaded, unchanged from http://javadesktop.org/jdic/demo/FileExplorer/fileexplorer.jar. ext.jar will contain our attack code. My first attempt used the fact that method look-up in Java follows inheritance, even for static methods. So all we need is a static initialiser in a class that extends the implementation main class. The application-desc tag will need to be fixed appropriately. class Attacker extends FileExplorer {     static {         ... do evil ...     }     // main of FileExplorer will be used. } If we start a thread up in the static block, we can execute after the real main has done its business. "Unfortunately," unlike the java command, javaws does not initialise the main-class. Next thing to try is replace a class the application expects to be part of it with our own code. It appears not to be (easily) possible to replace a class in the namespace (although I might have just messed that up). The third attempt relies on noticing that the application expects to be loading an extension. Referencing classes between packages is just what we need for this example to work. Using the standard suck it and see approach we see that the first class that needs to be present is org.jdesktop.jdic.desktop.DesktopException. So we add that: package org.jdesktop.jdic.desktop; public class DesktopException extends Throwable { } But this class does not get initialised. If we click on the Browse button, up crops org.jdesktop.jdic.desktop.Desktop. Excellent. package org.jdesktop.jdic.desktop; public class Desktop {     public static void browse(java.net.URL url) {     }     static {         // Demonstrate "l33t h4x0r skillz".         System.out.println(java.util.Arrays.asList(          [...]



Mixing trusted and untrusted code

Mon, 11 Dec 2006 18:55:48 -0500

I had thought Sun had realised mixing trusted and untrusted code was BAD. JNLP forbids it. The Java PlugIn has quietly added the restriction, apparently without telling anyone.

However, it appears that any code can be treated like an extension. So in order to mix all an attacker needs to do is pretend some of the code is an optional package. He or she can now make merry constructing stacks of trusted objects.

Have you effectively audited your signed code?




Who should care about mobile code security?

Sun, 10 Dec 2006 17:35:18 -0500

Mobile code security is a solved problem in Java, right? There are situations in which code must be secure. In particular code that is intended to be, or will ever be, used in: the Java library extensions containers for untrusted code (for instance, some web servers) libraries shared between contexts within secure containers trusted contexts within secure containers (for instance, signed applets) The first three bullets should, I hope, be pretty obvious. Shared libraries are slightly less obvious. It might seem self-evident that if you do not grant libraries any permissions then they'll be fine. However, careless use of statics can allow contexts to attack one another. The statics need not even by public or protected (wtf does anyone mean by "protected static"??). See User supplied objects may not behave as expected. Also reflection is open as, unsually, the security is based upon the immediate caller class, not the whole stack. The last bullet is presumably the most widespread problem. Any chump can sign an applet. He or she assumes that they are safe because Java is secure, as any fule no. Suppose an attacker wants to use some code signed by a trusted party. What are the things we, as honest programmers, need to be wary of? (Note that the signed code need not be current. It may be some old version stored away by the attacker for a rainy day, or found down the back of a sofa (signed code is more difficult to remove than chewing gum). So long as the certificates match, it does not matter if the code is from multiple releases. Each class file gets its own signature, so we can mix and match at that level.) The Java security model is predicated on the principle that if all the classes on the current thread's stack have a permission, then it is safe to exercise it[1]. However, state is also important. If untrusted code can construct an object structure that can execute a sensitive operation without any of the attack code on the stack, then it is on to a winner. For instance, perhaps in response to a button press by the user, a trusted action is performed using data read from some other visual components. Malicious code can make those components hidden and full of malicious data, then move the ActionListener from the button onto another innocuous looking button. Or just fire the listener with javax.swing.Timer - no button clicking necessary. ClassLoader prevents signed and unsigned code inhabiting the same package (although shared namespace is possible through multiple class loaders). So longs as nothing is public or protected and nothing extends a class from another package (other than from the Java library), we are in better shape. As it happens JNLP forbids applications from sharing signed and unsigned code. And Sun's applet implementation does the same, although I have not noticed that documented anywhere. So long as we assume Sun Microsystems' WebStart and PlugIn are the only trusters of signed Java code anywhere in the universe, it looks quite simple. We don't have to worry about malicious code fiddling with our signed works. Of course, we still have to worry about any data we use. If we follow commands from the server then we must make sure they are not malicious, for example. A big advantage that the Java library and extensions have over signed applications/applets is that they do not represent code for a functioning application. There simply is not code that adapts opening a file for use by main or a listener, say. JFileChooser does not itself read or write files. A little known feature of[...]



The curse of inheritance

Sun, 3 Dec 2006 15:29:20 -0500

We all know that implementation inheritance leads to fragility. If your class needs to be secure, then that goes double. Are you intending to override every inherited method? What if someone adds another, but doesn't bother to fix your class?

That happened with Java WebStart. FileOutputStream was subclassed to prevent annoyance code filling up your disk. After WebStart was written, NIO was added to Java. To support NIO, FileOuputStream adds extra methods. Annoying code can cast the supplied OuputStream to FileOutputStream and make merry with NIO. Incidentally, Socket streams are also implemented as FileXxxputStreams, though I have no idea what happens if you try to treat them as such.

The obvious solution is to implement as a smart proxy. Subtype the interface* type directly, forwarding onto the target object. This is particularly significant if the base type is serialisable.

*As in interface, not the Java keyword.




User supplied objects may not behave as expected

Sat, 25 Nov 2006 08:52:02 -0500

Apache Harmony had a little security flaw in its implementations of ThreadLocal. A field of Thread mapped ThreadLocals onto their thread-local values. However, it used Object.equals/hashCode to determine equality, not ==/System.identityHashCode.

The problem is that ThreadLocal can be supplied by untrusted code. Some malicious coder may decide it's a great idea to subclass ThreadLocal. Not to override initialValue, but equals and hashCode. Try initialising thread-local values four billion or so times (shouldn't take more than a couple of minutes), and you have yourself all the thread-local values for the current thread. Some of these values might be very interesting mutables.

The harmony source was corrected swiftly. Early versions of Sun's 1.3 suffered from the same problem, but I'm not sure if anyone ever noticed.

A similar problem occurred in snapshots of Sun JDK6. From 1.6, you can construct a String with a given Charset. A constructor that wont threaten to throw UnsupportedEncodingException. In some deterministic circumstances the character array used by the String is the same one wrapped in CharBuffer and passed to the potentially malicious Charset (actually CharsetDecoder). Evil Charset can keep hold of the CharBuffer (or the unwrapped character array) to mutate the String later.

The moral: User supplied object maliciousness may extend beyond data into implementation.

See:




Multicore processors can hide bugs

Fri, 24 Nov 2006 14:12:59 -0500

It is well known that bug dormant on single-threaded hardware become apparent on multiprocessor, multicore or simultaneous multithreaded systems. What I did not foresee was these systems hiding bugs that appear on simpler machines. With the rise of dual-core and SMT machines amongst developers, these bugs are increasingly likely to be overlooked until customers start to complain.

I came across the problem trying to run the GUI of a program that is supposed to find bugs. It appears that: A task is forked off in another thread, and a race started. In order not to throw an exception, the task must complete before a longer (but still short) task in the original thread. With multiple hardware threads, and on an otherwise idle system, this is easy - they run in different lanes and the fastest wins. On single-threaded hardware, because of the short time spans involved, it is unlikely that a thread switch will occur. So, the forking task usually travels all the way to completion before the forkee gets its boots on.

The morals: As well as multiprocessor testing, also test with a single effective thread. Keep watching your synchronizeds and volatiles.




Incorrect stack depth for security checks in java.util.concurrent.atomic.

Sun, 19 Nov 2006 18:04:14 -0500

Remember Bug 6243400? It's approaching the first anniversary of the Sun Alert Notification for this bug. I think that's time enough to be able to show it. My report is from March of last year and 1.5.0_04 was released just over three months later. Not sure why it has been changed from Bug to RFE. --- Note: you can send us updates about your Incident --- --- by replying to this mail. Place new information --- --- above these lines. Do not include attachments. --- --- Our system ignores attachments and anything below --- --- these lines. --- Hi Thomas Hawtin, We have determined that this report is an RFE and has been entered into our internal bug/RFE tracking system under Bug/RFE Id: 6243400 You can monitor this bug and look for related issues on The Java Developer Connection Bug Database at: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6243400 It may take a day or two before RFE shows up in this external database. The Java Developer Connection is a free channel that is maintained by staff here at Sun. Access this web page to join: http://developer.java.sun.com/servlet/RegistrationServlet. The home page for the Java Developer Connection is: http://java.sun.com/jdc. Thanks for taking the time to report this problem. Regards, Jitender ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ NOTICE: This message, including any attachments, is for the intended recipient(s) only. If you are not the intended recipient(s), please reply to the sender, delete this message, and refrain from disclosing, copying, or distributing this message. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ----------------- Report ------------------- category : java release : 5.0 subcategory : classes_util type : rfe synopsis : Incorrect stack depth for security checks in java.util.concurrent.atomic. description : FULL PRODUCT VERSION : java version "1.5.0" Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0-b64) Java HotSpot(TM) Client VM (build 1.5.0-b64, mixed mode, sharing) ADDITIONAL OS VERSION INFORMATION : Linux localhost.localdomain 2.4.20-8 #1 Thu Mar 13 17:54:28 EST 2003 i686 i686 i386 GNU/Linux A DESCRIPTION OF THE PROBLEM : java.util.concurrent.atomic.AtomicReferenceFieldUpdater and friends compromise security. AtomicReferenceFieldUpdater calls Class.getDeclaredField, which checks that the calling frame's ClassLoader is the same as that of the target class. As AtomicReferenceFieldUpdater is a boot class, any volatile reference of any boot class is accessible by user code, even in the presence of a security manager. From there it is trivial to escalate. You should note that there is a thread on the Mustang java.net forum that comes close to mentioning the issue. http://forums.java.net/jive/thread.jspa?messageID=12741&tstart=0#12741 STEPS TO FOLLOW TO REPRODUCE THE PROBLEM : Run the code below with 'java Elastica'. EXPECTED VERSUS ACTUAL BEHAVIOR : EXPECTED - An exception should be thrown. ACTUAL - The code executes without complaint. The String "/tmp" is printed as "/usr". REPRODUCIBILITY : This bug can be reproduced always. ---------- BEGIN SOURCE ---------- import java.lang.ref.SoftReference; import java.lang.reflect.*; import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;                                [...]



System.setSecurityManager(null);

Thu, 16 Nov 2006 09:05:37 -0500

Google Code Search or a plain old google will find plenty of hits. I find it difficult to imagine what is going through the mind of someone who would write such a thing.

System.setSecurityManager(null); removes all Java security from the entire process. Executed as an applet, any other applet running will have full access to the machine. Yay! Sun Java WebStart only runs one application per process, but other JNLP implementations share. It is also not unlikely that the code can be repurposed for contexts it was not designed to be used in.

ClassLoader imposes the restriction that classes from the same package (not necessarily namespace, mind) must be signed with the same certificate or not at all. The JNLP specification imposes a similar restriction across the application. The Sun Applet Plug-in appears to do something similar, but like so much Java security appears to be undocumented.

I chanced upon one rather entertaining piece of code. It cleared the security manager briefly, later attempting a restore. That wouldn't be so bad if Java didn't support exceptions or threads.




Sun Alert ID: 102622 and time to disclose

Wed, 15 Nov 2006 17:32:29 -0500

The timeline:

9 August 2005
Allude to the security vulnerability in e-mail to security-alert@sun.com.
27 December 2005
Noticing that it hasn't been fixed in u6, go into detail and provide example code.
12 August 2006 (according to freshmeat)
Patched JRE released (1.5.0 u8).
14 November 2006
Sun Alert ID: 102622 - A Security Vulnerability in the Java Runtime Environment Swing Library may Allow an Untrusted Applet to Access Data in Other Applets

Now it's not an Earth-shattering bug. Finding code worth attacking may be tricky. But is it just me who thinks the delay a tad long?

Never mind. As Java is becoming "free" as in RMS, anyone can create a fork to react in a sensible time period. No need for anything other than strict GPL. We don't want anyone else to benefit from our work, do we?




A serialisation optimisation for everyone

Mon, 13 Nov 2006 16:39:19 -0500

Here's a little optimisation. From a static initialiser, call ObjectStreamClass.lookup on all the classes you are going to serialise/deserialise. Assign the result to a static field. That's it.

Reflection only becomes efficient if you keep hold of the artifacts. Keeping hold of ClassLoader produce causes leaks. So, serialisation use a complicated soft reference cache.

If we keep hold of the artifact ourselves for the duration they are needed, then the cache will not miss. Reflection objects will not be recreated and compiled into the permanent generation. The permanent generation will not need long pauses for garbage collection.

I would be interested to hear how it works on real-life, large systems.




F3: Y?

Sun, 12 Nov 2006 18:57:15 -0500

F3 is a static-typed scripting language with type inference (in the mold of ML) that has been hitting the Sun weblogs. There is not much to impress so far. You'd think the introduction may start with something really tilted to unusual features of the language. To their credit, the first example isn't something that can't be written in a similar fashion in good old fashioned Java. The example is a (titleless) frame with button. And I thought Matisse was the r0x0rz for this stuff. You really want your library to support the style of your coding. Even without additional library support or Dolphin^W JDK7 languages features, this example is straightforward to transliterated into Java. I've left out that main and switching to EDT stuff. Only seems fair as the example assumes anything that takes more than one method isn't worth doing.   new JFrame() {{         add(new JButton("Press Me") {{               addActionListener(new ActionListener() {                   public void actionPerformed(ActionEvent event) {                     System.out.println("You pressed me");                   }               });         }});         setDefaultCloseOperation(DISPOSE_ON_CLOSE);         pack();         setVisible(true);     }}; With extended libraries and either a launcher or packager, the entire application could, in 100% Pure Java, become: import gui.*; class Example extends Frame {{     add(new Button("Press Me") {         public void onAction() {           System.out.println("You pressed me");         }     }); }} Is that better? No. The rest of the language The big, mad, stare-you-in-the-face problem with Java is the functions-are-for-wimps syntax. In F3 I don't even know where the semicolons go any more. They see to have been far too clever with the grammatical holes. It takes a giant leap backwards with header files. Wasn't that the original reason for Java? I assume they assume everyone will use a whirligig IDE. On a similar note, throwing objects that are not Throwable? The language appears to use arbitrary-precision integers. About time. On the other hand, implicit truncation. I can't see what it does for decimals. The variable cardinality looks as if it might be quite handy. I can't see how it handles the initial value problem. I hope it doesn't force assigning an error masking value that should never be used (fingers crossed). Language support for sequences (referred to as arrays or lists) is a sensible policy, but you should remember they are one of the three basic collection types: sequence, set and map. Treating them differently from one another, or attempting to merge them, results in disaster. Remember using Hashtables for Maps in 1.0/1.1? OCL seems to handle the three nicely. F3 is just a bit odd. And CAR/CDR? You do realise these refer to hardware details of the vacuum-tube IBM 704? How about head and tail, or learn first and butfirst from Logo? Triggers are not a nice part of SQ[...]



Serialisation optimisation for experts: Don't do it.

Thu, 9 Nov 2006 18:00:57 -0500

JavaLobby recently had a thread about Serialization and performance. Unfortunately the microbenchmark had some serious issues. So I have attempted to correct the issues, but have kept the original scope. This measures serialisation performance of very simple objects, that do not contain references to other objects (other than a String). The version of Pojo that attempted to define serialPersistentFields actually got the name wrong, so was the equivalent of the plain version. Perhaps there should be an annotation marking serialPersistentFields, serialVersionUID, readObject, writeObject, readResolve and writeReplace as members involved in the serialisation mechanism, similar to @Override. Correcting the name shows up that the primitive fields had been assigned wrapper classes. So, for instance, Float.class should become float.class/Float.TYPE. Another issue, if we are going to compare it to the later versions of the class, then we should really not share String instances. This can be achieved by appending the argument false to the ObjectStreamField constructors. Serializable objects that implement Externalizable do not have fields included in their class descriptors. Not a lot of people know that. The descriptions are used if there is no writeObject/readObject method and for defaultReadObject and readFields. So, for fairness, the fields for the writeObject/readObject version should be marked as transient. An important difference between the benchmarks is the contents of the class descriptor. The classes really shouldn't all have the same serialVersionUID as they are not all binary compatible. To see the impact this has, I have created a version of the driver that reuses the same stream for 100 objects, flushing after each. OriginalStream reuse AMD64 Server1.6.0-rc-b104x86 Client1.5.0_09-b01 Plain 20,769 ms35,598 ms9,224 ms serialPersistentFields21,640 ms36,199 ms12,589 ms writeObject/readObject14,547 ms22,077 ms9,589 ms Externalizable 14,806 ms21,999 ms9,190 ms [Update 11 Nov 06: I screwed up the last column. I had used ObjectOutputStream.writeObject to write each object instead of writeUnshared. So the object was written, followed by 99 references to the first. I have updated the table to use writeUnshared. I shall write my own microbenchmark and publish it under a free license.] [Update 31 Oct 07: I might get around to that sometime.] All tests were run unniced (su, nice -n -20) with 8 MB heap (-Xms8m -Xmx8m) on a Sun Ultra 20 with an Opteron 148 running Solaris 10. To get better figures, the test is run ten times as many as the original. I have not attempted to measure deserialisation performance. To summarise those figures: Don't use Externalizable Are all mixins a mistale? Do use streams for more than one object On the other hand, if you are not using unshared writes it will leak. Hand crufting writeObject/readObject can improve performance marginally. And may or may not improve binary compatibility maintenance. Not really important if you are going to write multiple instances of a class to the same stream. final fields become a pain. Run a modern JRE with the go faster button pushed If you can get people to stop using jview. Suck it and see... ... but under[...]



Closure mining: bottom up

Sun, 5 Nov 2006 20:14:16 -0500

While closures bring little more to Java than saner syntax, they do make practical different ways of doing things. So I wondered what sort of applications they would have straight in the Java library. It's a bottom up approach. You'd get laughed at if you tried to find patterns this way. But it's much easier to make up idioms this way. Attempting to trawl through some large "typical" source base sounds far too much like work. The categories I noticed:

Iteration over structures
iteration over key-value pairs, iteration with index, reducing, codepoint sequences, hierarchy traversals
Abstract Factory
generating the elements of a collection, ThreadLocal.initialValue
Transformation
transforming the elements of a collection, InheritableThreadLocal.childValue
Predicates
select matching, indexOf, validation
Simple event listeners
ActionListener, ChangeListener, PropertyChangeListener (if you must)
Execution in a different thread
EventQueue.invokeLater, DOMService.invokeLater [Common DOM API], Display.runSerially [MIDP], Executor.execute
try-finally
hands-dirty JDBC (big time), java.io (all over the place), locks, temporary context
Lock-free
do get ... while compare-and-set loops
JSP simple tags
add code within bodies (I'm not sure why inner classes are not mandated)
Combinations of the above.
iterate over results of a JDBC query

What have I missed? Can you suggest a minable source base that is "typical", high-level and open source?

Thinking about closures and SESE: are break, continue and premature return signs that the loop has not been abstracted?




Dynamically loading JDBC drivers

Tue, 31 Oct 2006 21:07:58 -0500

Apparently, sometimes it would be useful to be able to load JDBC drivers dynamically. Unfortunately, DriverManager will not allow access to drivers with classes that are not linkable from the calling class loader1. One solution is to hack the application class loader to add extra sources. There is an URLClassLoader.addURL[...]



Using Applets in place of JavaScript

Mon, 30 Oct 2006 16:48:07 -0500

Common DOM API is a little known feature of the Java Plug-In. The DOM containing the applet is available with interfaces to support reading, modifying and adding event listeners to the page. Everything we need to ditch JavaScript, right? I had known about the API for some time, but never gave it a spin. Nor had I ever seen any evidence of anyone else using it. Given the mess that is AJA"X", and JavaScript in general, this seems odd. Google Code Search finds one Hello World example, and that's about it. For my investigation I chose the most unoriginal, trashy-popular idea available: using Google Maps. This will involve some JavaScript communication too. First problem: The Common DOM API does not appear to support Opera. So, I installed Firefox 2.0. Attempting to clear the class loader cache from the Java console instead caused an NPE. Still, on my freebie 2GB Ultra-20 Opteron, restarting Firefox is relatively fast (faster than on my 256MB eSys Celeron, anyway). Getting hold of the root document is without entirely undue fuss. Access to the browser DOM should be confined to its thread, *apparently*. DOMAction is not generified, and the return value doesn't make sense when using invokeLater. (FWIW, I've come around to the point of view that JDK1.7 closures should probably modify locals of the enclosing method, rather than attempting to hack some result passed through return. It's awkward, it's a hack and it's irregular.) import com.sun.java.browser.dom.*; import org.w3c.dom.Document;     final DOMService service;     try {         service = DOMService.getService(applet);     } catch (DOMUnsupportedException exc) {         // Show something nice...         throw new Error(exc); // !!     }     service.invokeLater(new DOMAction/**/() {             public Object/*Void*/ run(DOMAccessor accessor) {                 Document doc = accessor.getDocument(applet);                 ...                 // Pointless return due to poor interface.                 return null;             }     }); Manipulating the DOM is much the same as normal. That is to say a bit of a pain as it's an API designed for dynamic languages. You have the added fun that half of the API turns out to be not implemented. import org.w3c.dom.*; import org.w3c.dom.css.*;     // The usual really DOM stuff.     // Add
    Element mapElement = doc.createElement("div");     mapElement.setAttribute("id", "map");     CSS2Properties mapStyle = (CSS2Properties)         ((ElementCSSInlineStyle)mapElement).getStyle();     mapStyle.setWidth("500px");     mapStyle.setHeight("300px");     body.appendChild(mapElement); So now onto interfacing to the Google Maps script and the wonder of netscape.javascript.JSObject (not specified in the JDK). Is[...]



Feeping Creaturism: New Fangled Closures vs Updated Local Classes

Sun, 27 Aug 2006 18:13:44 -0400

Neal Gafter has weblogged more about the closure proposal (Use cases for closures and Tennent's Correspondence Principle and Returning From a Closure). Function types Function types are an extreme extension to the Java type system. Client code isn't going to use them. Pointless. BTW: To be clear, I'm not suggesting using the most general of interfaces. I don't want to see types of the form interface ThreeArgsAndAReturn { RET runThisMethod(T0 arg0, T1 arg1, T2 args); }. I suggest interfaces allied to how the method will be used. The type documentation should be sufficiently concrete that the referring algorithm method @param should more or less specify just that it uses a particular type. Tennent's Correspondence Principle There are a couple of cases in which achieving Tennent's Correspondence with the closure proposal is not going to happen. Definite assignment goes out of the window. The exception simulating break, continue and return can be caught, or discarded by finally. A finally in an algorithm method wont be obvious to the application programmer. What is one to do? Discard exceptions caused by closing resources? Call std::abort? Should there not be a correspondence between closures and local classes? That's got to be confusing - having two mechanisms that the compiler treats differently when asked to make up your mind. Code that could equally appear in outer methods, closures and inner classes should always have the same interpretation. I'm sure some syntax for qualifying the nested instance or outer method is more than doable. Old ambiguous code can throw up warnings. We already see cases (that could appear in Java Puzzlers) where names referring to a different object within an inner classes cause problems (although static type checking helps). For instance, using this as a lock. To have code behave differently depending upon whether it was in a traditional inner class or a new fangled closure would be truly freaky. Using a special syntax for return from closure prevents Tennent's from working with nested closures. More puzzlers. Understanding There is agreeing something looks reasonable and there is understanding necessary to maintain and indeed create. Callbacks are an integral part of OO, but have poor understanding. How many programmers have problems grokking classic JSP tags? In my experience Java programmers don't get it (and neither do they get sequence diagrams). On the other hand, how many write rock solid BufferedInputStream code? Swing's AbstractDocument in at least one case doesn't use its own lock correctly, even though it is documented correctly. Hopefully this sort of thing will go someway to remove the easy-to-get-wrong code out of the hands of most programmers. Summary Most of the features come down to how much feeping creaturism you want. The complication of function types buys nowt. Having a duplicate feature with arbitrarily different disambiguation is absurd. [...]



The Closure proposal

Wed, 23 Aug 2006 17:32:07 -0400

I am going to assume you have seen the proposal. Essentially it is a parallel of anonymous inner classes with enhanced functionality. Developer vs library As with properties, the proposal addresses two sides which perhaps should be independent. On the application programmer side, we want to write functions in place with the minimum of effort. Probably the function will be passed to an algorithm method immediately. Terseness of definition is important. On the service provider side, as ever, the design should twist itself around providing something easier for the application programmer. It need not be terse, or have language features designed around it. The proposal seems to require a little duplication of information from what can be implied by use, on the client side. At the same time, the service provider side does not provide an interface that is as explicit or Javanic as it might be. If a library has a pair of algorithm methods that appear to require the same contract on a closure, then I need to compare item by item that the documentation is the same to ensure I can switch between the two. Lookup complexity vs explicit syntax One big change is that the namespace distinction between methods and variables/types is broken. This is a big change. Java doesn't have Koenig lookup, but it is already more than complicated enough. Perhaps a little extra syntax would be a better idea. To do perform an operation on an object, the identifier is usually followed by a dot: myVar.fn(). So one obvious form is to insert a dot to disambiguate. So instead of:     int result = fn(value); (fn could be a method or a variable), we have:     int result = fn.(value); A more extreme approach would be to remove the special syntax using generic base types.     int result = fn.invoke(value); Flow control How much more syntax do we want?! Rather obviously, it is useful to break out of loops, unless you are a SESE fanatic in which case you will need multiple functions (hey, perhaps they could be combined into a single type). A break out of a closure is effectively an exception throw. Perhaps instead of forcing the break syntax, exceptions could be made concise in this context. Labels could be used as checked exception type and instance. So:     Thing foundThing = null;     found: forEach(things, void(Thing thing) {             if (thing.isUseful()) {                 foundThing = thing;                 throw found;             }     }); The PDF doesn't explicitly mention anonymous functions and target typing for throws specifications, but they appear to be implied. Now if we go further and allow the algorithm method generic type to be inferred, we can have checked exception with a method declaration of:     public static void forEach(         Iterable items, void block(T item) throws EXC     ) throws EXC That will ensure that the exception cannot be thrown outside o[...]



An old favourite updated

Sat, 17 Jun 2006 09:42:41 -0400

Writing a program to print itself to standard output is a pointless task, but that isn't about to stop me. Printing the " character is the key to a concise solution. Various tricks make the program shorter. New in Java 5.0, PrintStream (and StringBuffer) catch up with Writer, by effectively allowing a print to combine with a substring/subSequence.

class T{static{String t="+(char)34;System.out.append(t+t+t,44,182);}}class T{static{String t="+(char)34;System.out.append(t+t+t,44,182);}}

See, tiger really does boost productivity.

"Thou shalt study thy libraries and strive not to reinvent them without cause, that thy code may be short and readable and thy days pleasant and productive."
-- The Ten Commandments for C Programmers, by Henry Spencer



Button responsiveness

Thu, 25 May 2006 13:28:20 -0400

Your average usability guideline requires action response time to have an upper limit of 50-100 ms (apart from web-usability guidelines which tend to state 5,000-10,000 ms). Responses that take more than 140 ms become disjoint - the impression of cause and effect is broken. Long response make the interface feel lathargic. It becomes irritating, and perhaps you start pushing the buttons harder.

One of the lunacies of most modern GUI toolkits is that mouse presses don't take effect until the button is released. Not intuitive and adds a pointless, arbitrary delay. How long is the delay? Here's an applet to play with (also available in a WebStart version). Dominant-hand, index-finger clicking isn't too bad, but remember the 50 ms limit for the complete operation.

I am not going to suggest you unilaterally confound the expectations of a few power users in order to improve the experience of most normal humans. Instead I suggest showing the result of the action on mouse down. On Swing replace buttonModel.isSelected() with buttonModel.isSelected() ^ buttonModel.isArmed().

Now try with other fingers. It's sluggish, no matter how hard you push.

Actions are more difficult than reflecting state. You will need some undo-like mechanism in case the user decides (or just accidentally) cancels the action. In general some other event could happen, possibly requiring the action to be undone, the event handled and then the action redone. If the action requires removing the source, you may want to commit early after a short timeout (as a pure guess, around 100 ms).

See:




Working around the ThreadLocal leak

Thu, 27 Apr 2006 18:53:16 -0400

A small performance regression is blocking the plugging of the ThreadLocal leaking. The google test indicates that the leak causes widespread pain. In discussion of the problem with Piotr Kobzda on comp.lang.java.programmer we wondered whether there was an easy-to-use workaround without modifying the boot classes. The answer is yes, I can do something other than whine about the JRE implementation. So here is my take on a drop-in ThreadLocal extension. The solution should be: compatible with ThreadLocal. Only the new/extends class name should change. It should be assignment compatible with java.lang.ThreadLocal, and allow methods, particularly initialValue, to be invoked and overridden as usual. comparable in performance to plain JRE 1.5/1.6 ThreadLocal. not leak thread-local values, even those that strongly reference the replacement thread-local. Values should become collectible in a prompt fashion. (Exact memory use deviates a little.) tolerant of class-loader garbage collection. The key idea is that the thread should only weakly reference the value, with a strong reference provided by the thread-local. For performance reasons, normal lookup should be performed without any threading code. Thread-safe code is necessary to update the strong referencing. Fortunately we can retain ThreadLocal, and its incestuous relationship with Thread, as the underlying mechanism. We need to place a WeakReference between the ThreadLocal.Entry and the actual value. This reference must not strongly reference non-boot class-loaders, so we cannot subclass. Therefore, we need to use ThreadLocal> as the implementation type yet provide a class that extends ThreadLocal. Inheritance is possible with fudging, but breaks initialValue. As WeakReferences are not updatable and we need something to keep strong references to the values, a holder object is inserted between WeakReference and value. The implementation ThreadLocal becomes ThreadLocal>. Assuming threads exit rarely, optimisation should be skewed heavily in favour of adding strong references rather than lookup or removal. This is easily achieved (from 1.5) through a standard lock-free approach. Removal of strong references on thread exit is more difficult than adding them. However, it need not be too aggressive, so we manipulate all but the first holder under synchronisation with a simple linear scan for invalidated holders. To detect thread exiting each thread-local contains a WeakReference for each thread it has been used with. The garbage collector adds the reference to its reference queue, which can be polled. The polling does mean the strongly referencable thread-locals may keep alive objects of dead threads. However, threads should cycle rarely, and using the thread-local in a new thread after a garbage collection should free the value. This reference can be combined with the holder object. As a slight enhancement, we can replace the holder dependence upon the Thread with a proxy. When a thread exits, its thread-local entries are cleared even if it is not its[...]



Do the funky FormLayout

Fri, 7 Apr 2006 19:10:01 -0400

Among the Java GUI cognoscenti FormLayout appears to be the non-JRE layout manager to talk about. As last time, the idea is to create the same simple layout. No cheating by going for an inappropriate design just because it makes the layout manager in question look better. There's the application window, and we want to layout the split pane's left hand side. Without further ado, the FormLayout 1.0.6 entry...         int gap = 2;         Size gapSize    = Sizes.pixel(gap    );         Size wideGapSize = Sizes.pixel(gap+gap); There's no happy middle ground with FormLayout. Either you use the mini-language, reminiscent of Perl, or use a slightly clumsy Java interface. We are not forced to use completely unnecessary and arcane literal strings.         // Button-labels top left.                // Text fields top right.                FormLayout layout = new FormLayout(             new ColumnSpec[] {                 new ColumnSpec(ColumnSpec.FILL, gapSize    , NO_GROW    ),                 new ColumnSpec(ColumnSpec.FILL, PREFERRED  , NO_GROW    ),                 new ColumnSpec(ColumnSpec.FILL, wideGapSize, NO_GROW    ),                 new ColumnSpec(ColumnSpec.FILL, PREFERRED  , DEFAULT_GROW),                 new ColumnSpec(ColumnSpec.FILL, gapSize    , NO_GROW    ),             },             new RowSpec[] { }         ); I guess if I was serious about FormLayout, I'd shove in a ColumnSpec-sequence builder or a broken down appendColumn method. Note, both ColumnSpec and RowSpec have a FILL constant with the same type, so import static to get rid of the duplication is not a good idea.         JPanel pane = new JPanel(layout);         CellConstraints cons = new CellConstraints();                 layout.appendRow(new RowSpec(gapSize));         for (String key : JobserveMeta.instance.getSpecialFields()) {             // Buttons are a bit tall, so we don't use a gap row.             layout.appendRow(new RowSpec(PREFERRED));             pane.add(                 view.createKillButton(key),                 cons.xywh(2, layout.getRowCount(), 1, 1)             ); Just to confuse, FormLayout uses FORTRAN-style, 1-based indexes. That can be quite handy as it doesn't require either,[...]



Distinctly GroupLayout

Wed, 5 Apr 2006 04:19:21 -0400

It was designed as a very flexible layout manager for GUI builders. To be simple and intuitive. However, SpringLayout (SL) never took off. Now comes GroupLayout (GL), kind of like GridBagLayout (GBL) without the grid coordinates. For a GL tutorial, see Tomas Pavek's Blog. A couple of years ago, for comparison, I converted a simple layout from GBL to SL. The effort succeeded in the sense that after some considerable hair tearing (not in picture top right), I came up with some functioning code. GBL is not nice to use raw; SL is outright infeasible. For both layout managers, I wrote some code to smooth the path. So now is time to dust the example down, and apply the all singing, all dancing, shiny-new GL. The area in question is left of the split-pane, containing the four pairs of buttons and fields, and the text-area beneath. For comparison, the original GBL code. The important variable not declared here is view, which creates the relevant components.         // Button-labels top left.                // Buttons are a bit tall, so other than first don't inset.         Constraints labelCons = new Constraints()             .gridLocation(0, 0)             .anchor(WEST)             .fill(HORIZONTAL)             .insets(inset, inset, 0, inset);        // Text fields top right.         Constraints fieldCons = new Constraints()             .gridLocation(1, 0)             .weight(1.0, 0.0)             .fill(HORIZONTAL)             .insets(inset);         // Main pane underneath.         Constraints mainCons = new Constraints()             .gridLocation(0, 0)             .gridSize(2, 1)             .weight(1.0, 1.0)             .fill(BOTH)             .insets(inset, 0, 0, 0); // Bottom field button label inset.         GridY gridY = new GridY(labelCons, fieldCons, mainCons);                     JPanel pane = new JPanel(new java.awt.GridBagLayout());         for (String key : JobserveMeta.instance.getSpecialFields()) {             labelCons.add(pane, view.createKillButton(key));             fieldCons.add(pane, view.createField(key));             gridY.inc();             // Clear inset from top line.             labelCons.insets(0, inset);         }         mainCons.add(pane, createMainScrollPa[...]



Hash imperfection

Wed, 1 Mar 2006 18:33:17 -0500

How difficult can hash algorithms be? Take some values and produce another value. Any value. Recently I have stumbled across two poorly performing hash algorithms.

One was in javac. In com.sun.tools.javac.util.Name to be precise. Name.hashValue ignores all but the first, middle and last characters (actually bytes). As is usual with these things, the entire sequence will either be compared or copied at more-or-less the same time. The data will usually be cached anyway. In order to achieve a negligible gain in performance when computing the hash code, clashes often occur between similar names.

The consequence of a hash gone wrong are severe. Instead of the legendary hash table O(1) performance, suddenly it is O(n). For n accesses, O(n2) replaces O(n). Oops. Problems that should be I/O-bound suddenly become CPU-bound.

In the case of javac synthetic names are likely to tickle this problem. I came across it trying to measure the overhead of using many small anonymous inner classes (about 1K each). The sophisticated performance profiling technique necessary to identify something so subtle: ctrl-\.

Another algorithm I saw recently almost got it right. Multiply values of a sequence by powers of 31, and then sum. The first value is multiplied by 25-1; the second by 210-26+1; the third 215-211-210+26+25-1; the fourth 220-217+212+211-27+1. Well, you see where the low order bits are going. Not good.

Back in Java 1.0 and 1.1, String made a less severe sampling "optimisation" similar to javac names. Thankfully that got fixed to something sane. A useful source for inspiration. Just don't try to be too clever.

See:




Another nail in the coffin of Thread.suspend.

Tue, 21 Feb 2006 16:02:53 -0500

Thread.suspend has been deprecated since 1.1, for good reason. "Thread.suspend is inherently deadlock-prone. If the target thread holds a lock on the monitor protecting a critical system resource when it is suspended, no thread can access this resource until the target thread is resumed. If the thread that would resume the target thread attempts to lock this monitor prior to calling resume, deadlock results. Such deadlocks typically manifest themselves as "frozen" processes." -- Why Are Thread.stop, Thread.suspend, Thread.resume and Runtime.runFinalizersOnExit Deprecated? The vast majority of threaded code will hold a lock at some point. The lock-free style seems very rare (unsurprisingly). Even calling Component.repaint involves a lock. A lock that is held for a very short time, say writing a single reference, is unlikely to be caught out by a Thread.suspend. You may well get away with it. java.util.concurrent.locks.ReentrantLock behaves differently from the Java language lock. A thread blocking on a Java language lock cannot succeed while suspended. However, a thread blocking on a ReentrantLock prevents other threads acquiring that lock out of turn. For some locks, certain threads may spend a significant proportion of time waiting to acquire. If those threads are suspended, switching to a ReentrantLock implementation could spell deadlock. The NervousText JDK demo uses a thread that periodically requests a repaint. Back in JDK 1.0, the thread was suspended with Thread.suspend. In practice that actually works adequately even today. The use of Thread.suspend seems to appeal to programmers who don't like the over complicated usage of locks. Some (most?) programmers prefer to draw immediately rather than queue a repaint (bad choice). Java SE 6 XAWT uses ReentrantLock in place of a Java language lock. This lock is highly contended. It appears that in some situations there is a high chance of a Thread.suspend assisted deadlock. In particular, under the applet viewer, a suspend of a painting thread in response to a GUI event often causes a hang. For instance, on the congratulations screen of Darren Salt's Advanced Lawnmower Simulator, switching between it and another window tickles the problem on occasion. Hung from Sun claims not to be able to reproduce the problem, but I'm not sure what they are doing. If you have old code hanging around that still uses Thread.suspend, you might find curious undetectable deadlocks as java.util.concurrent makes headway. It may well pay to make the code safe now. See: Bug 6317330: Use ReentrantLock to serialize access to Xlib, etc for improved performance [...]



Reflection overreach

Sat, 11 Feb 2006 19:09:22 -0500

Two months ago I was on about how reflection in trusted code can cause security flaws. I stated, "A conservative policy is to only allow access to public items." That is not necessarily conservative enough. I noticed the other day that you should not be able to access sun.* packages (and in WebStart com.sun.javaws.*, com.sun.deploy.* and org.mozilla.jss.*). If your code is trusted, then SecurityManager.checkPackageAccess appears to be the method you need to call to check whether a package is on the blacklist. It's not just for class-loaders (see the URLClassLoader.newInstance source).

Reflection's use of only the direct caller to determine access is unorthodox. Any reflection done by library code on behalf of client code will have the wrong caller. Both class-loader and package will be wrong. So be very careful with client code supplied methods and method names (and fields/constructors/classes). To shed privileges you can call reflection from code loaded by an untrusted class-loader or use reflection to invoke reflection.

In addition to getting hold of accessible objects (methods, constructors and fields), access is also checked on invoke/get/set. That is, unless setAccessible is set. Exactly what is supposed to happen is very poorly documented. If you want to run fast and loose, setAccessible seems the way to go even when it is unnecessary.

When problems with the use of reflection were discovered, you'd think the relevant authorities at Sun would make sure they corrected everything. Or perhaps even when they took a second bite at the cherry. Ho hum. As for mustang, the security bug I submitted is still present in b71, around a year after it was first discovered.

It does not come as a surprise to me that complicated systems have bugs, and security is not immune from those bugs. The events since last March leave the notion that Java is naturally secure in tatters. Checking array bounds is not enough. Anyone for a new language/VM/library combo?

See:




Regressing mustangs

Mon, 6 Feb 2006 21:58:07 -0500

It appears that I have the first non-duplicate, non-Hello World(s)! entry to appear in the Project Mustang Regression Challenge. Okay, it's on the contrived side. Looking through the diffs I found something while still at java.a.

A much more useful and easier approach is to run your own applications and test suites on it. If your app gets screwed up when Java SE 6 is launched, then it's your fault for not having the fix made. Of course, the exercise may also throw up bugs in your coding.

On the other hand, I'd quite like to replace my £155 (inc VAT) thing with a more upmarket Ultra 20.

See:




Detecting invokeAndWait abuse

Thu, 19 Jan 2006 20:05:06 -0500

An apparently lesser known fact about EventQueue.invokeAndWait is that it can cause deadlocks. I suppose the reasons are a little subtle, but the result is not. Your desktop might not be going to Pluto, but there's no need to be all Mickey Mouse about it. Just the other day JEdit hung on me. I took jstack to it, but there were no deadlocks detected. There was, however, a use of invokeAndWait. Sure enough the EDT was waiting on monitor locked by the thread calling invokeAndWait. See the bug report for the full trace. In this case it is a different method holds the lock from the one that called invokeAndWait. A neat example. One I could show as an example of why you need to be careful with invokeAndWait. While checking for references, I found the old How to Use Threads in Swing Tutorial mentions "make sure that the thread that calls invokeAndWait does not hold any locks that other threads might need while the call is occurring." That makes weblogging the issue much less sexy. To add value I thought it might be nice to write a bit of code to detect holding any locks while invokeAndWait is called. Writing correct application code in the first place would be better, but you can test instead. Recent builds of Mustang expose held locks through a JMX interface. That will do for our purposes. It's worth testing applications on early release versions anyway, as it can make certain difficult errors more apparent. Again in JEdit a deadlock associated with setCursor happened reproducibly under early Tiger releases. My first thought was to hack EventQueue and use -Xbootclasspath/p:. Then it occurred to me that we can instead insert a subclassed EventQueue. That can be done from a Java agent, so no need to play around with the bootclasspath or with the application. Security managers and the multiple EDTs of applets are not catered for. public class InvokeAndDeadlock {     public static void premain(         String options, java.lang.instrument.Instrumentation inst     ) {         WarningQueue.install();     } } import java.lang.management.*; public class WarningQueue extends java.awt.EventQueue {     /** The top few frames will just be us, and so are not of interest. */     private static final int IGNORED_FRAMES = 5;     private static final ThreadMXBean threadBean =         ManagementFactory.getThreadMXBean();             public static void install() {         java.awt.EventQueue oldQueue =             java.awt.Toolkit.getDefaultToolkit().getSystemEventQueue();         oldQueue.push(new WarningQueue());     }     @Override     public void postEvent(java.awt.AWTEvent event) {         if (event instanceof java.aw[...]



On while-free waits

Sat, 24 Dec 2005 12:10:54 -0500

In Java 1.5, the Object.wait documentation was "clarified" to permit spurious wakeups. This means you really do have to put wait within a while loop. Having said that, Sun's JVM does not currently wake spuriously. There is a technique that involves wait outside of a while loop. I though it was a bug the first time I saw it, but it does work in the absence of spurious wakeups. If you see it, it should be fixed. Suppose we want to perform a task on another thread. We can create a private lock. Hold the lock while placing the task on a queue, and then wait. That task will notify the lock when finished. Because we queued the task with the lock held, the notify cannot slip by before our wait. As the lock is private nothing else can notify on the lock. class Lock { } final Object lock = new Lock(); Runnable task = Runnable() { public void run() {         try {             ... do some stuff ...         } finally {             synchronized (lock) {                 lock.notify();             }         } }}; // !!! DO NOT DO THIS !!! synchronized (lock) {     // Because we are synchronised to lock here,     //  notifyAll will not execute until lock is released by wait.     blockingQueue.put(task);     lock.wait(); } Clearly spurious wakeups destroy the technique. A correction is straightforward enough. Introduce a flag. Wait while the flag is not set. Set the flag when done, together with the notify. class Lock { } final Object lock = new Lock(); class Task implements Runnable() {     boolean done;     public void run() {         try {             ... do some stuff ...         } finally {             synchronized (lock) {                 done = true;                 lock.notify();             }         }     } } Task task = new Task(); blockingQueue.put(task); synchronized (lock) {     // Because we are synchronised to lock here,     //  notify will not execute until lock is released by wait.     while (!tast.done) {         lock.wait();     } } java.awt.event.InvocationEvent offers to notifyAll when the invocation has completed. However, there is no way to check if the wake was from the task exiting, or if it was just spurious. Fortunately, we can subclass and rescue the situation. The dispatch method can be overridden to do the obvious. class SafeInvocationEvent extends InvocationEvent {[...]



Erase reference arrays

Fri, 9 Dec 2005 00:18:08 -0500

In the days before 1.5, arrays offered a poor person's generics. Arrays do not play nicely with generics. Old ways need to be updated.

It used to be common practice to return from methods an array rather than some form of Collection. With an array you can state the type of the contents within the programming language. The interface is more readily understandable. The client code is often less messy, as it needs no casts. If a collection is returned, the type documentation must be moved to somewhere within the JavaDocs text. Unfortunately, arrays are less flexibility than collections, but that is a price most find worth paying.

With generics, reference arrays have become marginalised. Arrays are no longer required for typed collections. Collection or List can be returned with pretty much all the advantages of arrays and the flexibility of collections. We just have to make sure to make that defensive copy. There should be very little effect on performance in most cases.

However, use of arrays appears to have become ingrained. Arrays have become the first choice when a collection is required. Even if instances are kept within a class, the typing and concise syntax can often make the code clearer.

If we want an array of a generic type, we are in trouble. List[] is prohibited. (List)new List[] is compilable, but produces an unchecked warning. Any code (give or take a few errors) that could create mysterious ClassCastExeptions generates warnings. List[] is assignable to Object[], but then we do not get an ArrayStoreException when a List is stored in it.

If the code has lint, remove it. Do not suppress compiler help, unless you have an overriding need to do so.

The easy solution: Drop arrays, and stick with generic collections. Just create a List>() or whatever, and be done.




Sleeper Security

Wed, 7 Dec 2005 14:42:34 -0500

I personally find security dull. However, most Java programmers need to be aware of how to avoid inadvertently opening up a security flaw. Even the best fail. The general approach to security checks in Java is to examine the privileges of frames on the stack from the top downwards. Each method involved must have the requested privilege. That is, the intersection of rights is considered. For convenience and performance, AccessController.doPrivileged allows privileged code to run with underprivileged frames below. The reflection API takes a modified approach. The default policy is to allow code access to private variables only of classes loaded by the same class loader (see SecurityManager.checkMemberAccess). Examining the caller of a method is highly unusual in Java. One side-effect is that non-application code using reflection has to be extremely careful with the arguments it passes on. A conservative policy is to only allow access to public items. By non-application code I mean any code that might now or in the future be accessible by code from other class loaders. For instance, libraries that are shared between web apps in a web application server. An example of this problem is the java.util.concurrent.atomic field updater classes. They, very usefully, support a set of atomic operation on volatile fields. You could not ask for an author more careful or better able to deal with mind-twisting complexity. Perhaps that means there are few people with the balls to consider questioning the code. The only check, prior to 1.5.0 update 4, that the field updaters do is to ensure that the field in question is volatile. The Java library classes are all loaded by the bootstrap class loader. Under the default security policy this means users of this class cannot use it to update fields of their own code. However, any method can now fiddle with any volatile field of a Java library class. For instance, volatile fields are used by the security caches within the reflection implementation. Bugs like this can affect applets, JNLP/Web Start applications and anything else that executes untrusted code. Web application server instances that attempt to run web apps from multiple customers are affected. RMI clients can dynamically load code. Jini is built on RMI. I independently discovered the concurrent.atomic bug after reading a posting by the mysterious subanark. My report was dated just over three months before the fix was released (Sun Mar 20 17:03:11 MST 2005, RFE Id: 6243400). Mistakenly, I called the demonstration code Elastica, but they didn't cover Atomic. It appears that Adam Gowdiak found this particular bug (among others) before me. In my opinion, taking over three months to fix such a critical security bug is outrageous. Fixing these sort of problems should not to be tied to updates of minor,[...]



Stop the EDT, I want to get off.

Sun, 27 Nov 2005 19:29:55 -0500

It used to be considered the height of sophistication for Swing code to eagerly construct GUI components in the background, so that they could be displayed without delay when required. The argument that it takes significant time to construct components is no longer valid. You may still wish to load any external data in the background. Even do some work between mouse down and mouse pressed. In the age of well optimised JVMs and desktop processors with multiple hardware threads, threading issues are more important than ever. Swing is poorly written and even thread-hostile in places. The real killer is the hidden use of invokeLater. There you are keeping the EDT at arms length, and somewhere in the depths of Swing it all goes multi-threaded. The code below shouldn't even kick off the EDT, but that is not what happens. import javax.swing.event.DocumentEvent; import javax.swing.text.StyledDocument; public class PoorCode {     public static void main(String args[]) {         javax.swing.JEditorPane pane = new javax.swing.JEditorPane();         pane.setContentType("text/html");         pane.setText("Oops");         StyledDocument doc = (StyledDocument)pane.getDocument();         doc.addDocumentListener(new javax.swing.event.DocumentListener() {                 public void changedUpdate(DocumentEvent event) {                     if (java.awt.EventQueue.isDispatchThread()) {                         System.err.println("EDT!!");                     }                 }                 public void insertUpdate(DocumentEvent event) {                 }                 public void removeUpdate(DocumentEvent event) {                 }         });         javax.swing.text.Style style = doc.addStyle("style-name", null);         style.addAttribute("attribute-name", "attribute-value");     } } Summary: Keep all Swing component manipulation on the EDT straight and narrow. See also: Debugging Swing - is it really difficult ? by Alexander Potochkin [...]



Counting finalised objects

Tue, 11 Oct 2005 17:37:45 -0400

Churn of finalisable object is not a good thing, for reasons I am sure you are familiar with. I am sure there is an off-the-shelf way of finding out how frequently your code finalises. However, all the Sun JVM seems to provide is the number of pending finalisable objects, with peek if you are lucky. It should be easy to check when objects are finalised and periodically dump statistics. So I did. java.lang.ref.Finalizer runs finalisers when their time has come. So we can alter this class to take measurements. Putting -Xbootclasspath/p:classes on the command line will load our modification in preference to the code in rt.jar. I am using Java SE 6 (ea b55), but J2SE 5.0 should do just as well. Modifying code before the library is initialised is full of problems, but we can make do. In the Finalizer.runFinalizer method, just after the call to invokeFinalize, we have an opportunity to take a note of the class of the object that has just been finalised. Just the name should do, which has the advantage of not keeping a string reference to the Class around and is trivial to sort. We don't really care about class loaders. Insert this code.                 // Collect stats, if may be wanted.                 if (collectStats != Boolean.FALSE) {                     String name = finalizee.getClass().getName();                     finalizee = null;                     synchronized (countMap) {                         Counter count = countMap.get(name);                         if (count == null) {                             count = new Counter();                             countMap.put(name, count);                         }                         count.increment();                     }                 } else Somewhere in the Finalizer class, we will need to put the map, and code to dump the statistics. All straight forward stuff.     /**     * Keeps a simple count     * Count starts from zero.     */     private static class Counter {         /**         * Cur[...]



Trig happy

Tue, 2 Aug 2005 19:32:15 -0400

Java goes to great lengths to ensure that when you use a trig function the argument is normalised with a very precise value of π. However if you look at Math/StrictMath toRadians or toDegrees you'll witness the total absence of any attempt at accuracy. double standards?