Source Maps from multiple files using AjaxMin.dll

Jun 10, 2013 at 9:01 PM
I a trying to generate a correct source maps file when minifying multiple files using the AjaxMin.dll, but the generated output is never correct when I try to consume it. Specifically the line numbers seem to generally target line 1 of the last file no matter where to error is.

This is probably just me doing it wrong especially since Minifier.MinifyJavaScript seems to target a call using just a single JavaScript file (which works just fine).

If more custom code is needed could you please help by pointing me in the right direction?
Thank you.

example
a.js
b.js
c.js

should result in

combined.js & combines.js.maps
Coordinator
Jun 10, 2013 at 9:11 PM
Not sure I follow you. Are you saying you are minifying a.js, then b.js, then c.js separately with AjaxMin, and then trying to create a combined.js and combined.js.maps with custom code? Or are you using AjaxMin to combine, minify, and map a.js, b.js, and c.js together? If it's the former, does running a.js, b.js, and c.js though the AjaxMin command-line tool generate the proper map? If not, do you have an example set of input files I can debug?
ajaxMin a.js b.js c.js -o combined.js -map:v3 combined.js.maps
Jun 10, 2013 at 10:05 PM
Thank you for taking the time.

The command line tools produces the correct combined.js & combined.js.map

I have reduced the code to the following where I am trying to map the error thrown by "UUUUUU" in b.js

a.js
define("jake");
b.js
define("boo");

console.log(123);

UUUUUU
c.js
define("coo");

function define() { }
Using the commandline tool I get what seems a correct output.

combined.js
function define(){}define("jake"),define("boo"),console.log(123),UUUUUU,define("coo")
/*
//@ sourceMappingURL=combined.js.map
*/
combined.js.map
{
"version":3,
"file":"combined.js",
"lineCount":1,
"mappings":"AAEAA,SAASA,MAAM,CAAA,CAAG,ECFlBA,MAAM,CAAC,MAAD,CAAQ,CCAdA,MAAM,CAAC,KAAD,CAAO,CAEbC,OAAOC,IAAI,CAAC,GAAD,CAAK,CAEhBC,MACA,CFLAH,MAAM,CAAC,KAAD",
"sources":["c.js","a.js","b.js"],
"names":["define","console","log","UUUUUU"]
}
However when I use the Minifier object I am uncertain how i will get this same result. One approach I have tried that does not work is like the following
var sw = new StreamWriter(new FileStream("combined.js.map", FileMode.Create), new UTF8Encoding(false));

var sourcemap = new V3SourceMap(sw);

var settings = new CodeSettings { 
    SymbolsMap = sourcemap
};

var minifier = new Minifier();

var sb = new StringBuilder();
foreach (var file in list)
{
    minifier.FileName = file.Name;
    sb.Append(minifier.MinifyJavaScript(file.Content, settings));
}

var output = sb.ToString();
The above yields the following result

combined.js (which is very invalid since I am simply concatenating the results together)
define("jake")define("boo"),console.log(123),UUUUUUfunction define(){}define("coo")
combined.js.map
{
"version":3,
"file":"",
"lineCount":1,
"mappings":"AAAAA,MAAM,CAAC,MAAD,bCANA,MAAM,CAAC,KAAD,CAAO,CAEbC,OAAOC,IAAI,CAAC,GAAD,CAAK,CAEhBC,/BCFAH,SAASA,MAAM,CAAA,CAAG,EAFlBA,MAAM,CAAC,KAAD",
"sources":["a.js","b.js","c.js"],
"names":["define","console","log","UUUUUU"]
}
So as you can see I have a few problems. Firstly I am not sure how to work with the Minifier.MinyJavaScript method since it returns "crunched" JS for each individual file while only creating 1 .map file.
Secondly, if I combined all the JavaScript before I call MinifyJavaScript then how do I control how the minifer knows where each file starts and ends?

I hope it is clear what I am trying to do.

Thank you.
Coordinator
Jun 11, 2013 at 12:54 AM
Okay, I think I understand what you are doing now. First off, to address the invalid concatenated output, you need to set the TermSemicolon property on your CodeSettings object to true. That will ensure that every segment of minified JS you create will be properly terminated so subsequent concatenations will be valid.

Now, as for the symbol map output, definitely not a use-case scenario I had considered. You're reusing the same V3SourceMap object for multiple calls to the MinifyJavaScript method. I'm kinda surprised it even produces output at all, really, but I definitely see the value in being able to do this. I'll look into what it would take to make this scenario work in the first place.

The way I would've done it is combine the three input files into a single string, then run the combined string though MinifyJavaScript to produce the map file. Of course, I also happen to know about the undocumented ///#SOURCE directive, which is how this is all done in the EXE and build tasks. Whenever the scanner encounters a ///#SOURCE comment (format below), it parses the following source as if it came from the given source file starting at the given position. The format of the comment is:
///#SOURCE line col path
Line and Col are both numeric values, and path is everything after the whitespace following the column to the end of the line. The line immediately after the comment is treated as if it came from the file path startling at line line (1-based) and column col (also 1-based). So if you were to combine your input files, putting the appropriate ///#SOURCE directives before each file, you would get source code that looks like this:
;///#SOURCE 1 1 a.js
define("jake");
;///#SOURCE 1 1 b.js
define("boo");

console.log(123);

UUUUUU
;///#SOURCE 1 1 c.js
define("coo");

function define() { }
And when you minify that code, the output and the symbol file should all work properly. Notice something else I did: each ///#SOURCE directive starts on its own line, and begins with a semicolon before the actual comment starts. If you are concatenating distinct JS modules, it's a good practice to follow, just to protect against those modules that think they don't need to properly terminate.
Jun 11, 2013 at 4:12 PM
Thank you very much!

It works perfectly using ";///#SOURCE line col path" and building the entire file before minifying it. This actually serves my purpose better since I do not need each individual file minified but instead actually want all possible minifications applied in its entirety.

I have a followup question regarding the "line" and "col" part of the ///#SOURCE directive.
In what scenarios would these have different values than 1 and 1?

Regarding how I first tried using the minifier object, I believe that I was actually using it more like a "builder" object in which case I would expect the "MinifyJavaScript" method to be of type void and instead expose the final result through ex. "ToString". But as I see now this would be a different object and workflow.

Once again thank you.
Coordinator
Jun 11, 2013 at 4:41 PM
If you were pulling the script from inside a <SCRIPT> element within an HTML file, then the line and col would not be "1 1" -- they would correspond to the actual line and character of the script inside the <SCRIPT> element. If you are simply using stand-alone files, then no -- they should always be "1 1."
Coordinator
Jun 12, 2013 at 8:04 PM
Say, I wanted to see why your original approach wasn't working, so I wrote a quick app based on your previous posts that minifies those three files above separately as c.js + a.js + b.js while using the same symbol map object:
        static void Main(string[] args)
        {
            var inputFiles = new string[] { "c.js", "a.js", "b.js" };
            var mapPath = "combined.js.map";
            var outputPath = "combined.js";

            var utf8 = new UTF8Encoding(false);
            var outputBuilder = new StringBuilder();
            using (var outputWriter = new StringWriter(outputBuilder))
            {
                using (var mapWriter = new StreamWriter(mapPath, false, utf8))
                {
                    using (var sourceMap = new V3SourceMap(mapWriter))
                    {
                        sourceMap.StartPackage(outputPath, mapPath);
                        var settings = new CodeSettings
                            {
                                SymbolsMap = sourceMap,
                                TermSemicolons = true
                            };

                        var minifier = new Minifier();
                        foreach (var filePath in inputFiles)
                        {
                            minifier.FileName = filePath;
                            outputWriter.Write(minifier.MinifyJavaScript(File.ReadAllText(filePath), settings));
                        }

                        sourceMap.EndPackage();
                        sourceMap.EndFile(outputWriter, "\r\n");
                    }
                }
            }

            var code = outputBuilder.ToString();
            Debug.WriteLine(code);
            File.WriteAllText(outputPath, code, utf8);

            Decoder.Decode(mapPath);
        }
The resulting output was:
function define(){}define("coo");define("jake");define("boo"),console.log(123),UUUUUU;
/*
//@ sourceMappingURL=combined.js.map
*/
And the map file is:
{
"version":3,
"file":"combined.js",
"lineCount":1,
"mappings":"AAEAA,SAASA,MAAM,CAAA,CAAG,EAFlBA,MAAM,CAAC,KAAD,CAAO,hCCAbA,MAAM,CAAC,MAAD,CAAQ,dCAdA,MAAM,CAAC,KAAD,CAAO,CAEbC,OAAOC,IAAI,CAAC,GAAD,CAAK,CAEhBC,M",
"sources":["c.js","a.js","b.js"],
"names":["define","console","log","UUUUUU"]
}
I won't include the Decoder source here (it's a lot of code), but the end result sure looks like all three source files are encoded into the resulting map file:
Name: version, Value: 3
Name: file, Value: "combined.js"
Name: lineCount, Value: 1
Name: mappings, Value: "AAEAA,SAASA,MAAM,CAAA,CAAG,EAFlBA,MAAM,CAAC,KAAD,CAAO,hCCAbA,MAAM,CAAC,MAAD,CAAQ,dCAdA,MAAM,CAAC,KAAD,CAAO,CAEbC,OAAOC,IAAI,CAAC,GAAD,CAAK,CAEhBC,M"
Name: sources, Value: ["c.js","a.js","b.js"]
Name: names, Value: ["define","console","log","UUUUUU"]
AAEAA   (0,0,2,0,0)     min:(1,1)       src: (3,1)       name:'define'          c.js
SAASA   (9,0,0,9,0)     min:(1,10)      src: (3,10)      name:'define'          
MAAM    (6,0,0,6)       min:(1,16)      src: (3,16)                             
CAAA    (1,0,0,0)       min:(1,17)      src: (3,16)                             
CAAG    (1,0,0,3)       min:(1,18)      src: (3,19)                             
EAFlBA  (2,0,-2,-18,0)  min:(1,20)      src: (1,1)       name:'define'          
MAAM    (6,0,0,6)       min:(1,26)      src: (1,7)                              
CAAC    (1,0,0,1)       min:(1,27)      src: (1,8)                              
KAAD    (5,0,0,-1)      min:(1,32)      src: (1,7)                              
CAAO    (1,0,0,7)       min:(1,33)      src: (1,14)                             
hCCAbA  (-32,1,0,-13,0) min:(1,1)       src: (1,1)       name:'define'          a.js
MAAM    (6,0,0,6)       min:(1,7)       src: (1,7)                              
CAAC    (1,0,0,1)       min:(1,8)       src: (1,8)                              
MAAD    (6,0,0,-1)      min:(1,14)      src: (1,7)                              
CAAQ    (1,0,0,8)       min:(1,15)      src: (1,15)                             
dCAdA   (-14,1,0,-14,0) min:(1,1)       src: (1,1)       name:'define'          b.js
MAAM    (6,0,0,6)       min:(1,7)       src: (1,7)                              
CAAC    (1,0,0,1)       min:(1,8)       src: (1,8)                              
KAAD    (5,0,0,-1)      min:(1,13)      src: (1,7)                              
CAAO    (1,0,0,7)       min:(1,14)      src: (1,14)                             
CAEbC   (1,0,2,-13,1)   min:(1,15)      src: (3,1)       name:'console'         
OAAOC   (7,0,0,7,1)     min:(1,22)      src: (3,8)       name:'log'             
IAAI    (4,0,0,4)       min:(1,26)      src: (3,12)                             
CAAC    (1,0,0,1)       min:(1,27)      src: (3,13)                             
GAAD    (3,0,0,-1)      min:(1,30)      src: (3,12)                             
CAAK    (1,0,0,5)       min:(1,31)      src: (3,17)                             
CAEhBC  (1,0,2,-16,1)   min:(1,32)      src: (5,1)       name:'UUUUUU'          
M       (6)             min:(1,38)      
So it looks like it should work the way you were doing it. Do you see anything I'm doing in this code that you weren't doing in yours?
Jun 15, 2013 at 9:41 PM
Edited Jun 15, 2013 at 9:45 PM
So I finally got around to testing this out. As you correctly state, on the face of it the source map seems right. But it seems to be the case that errors are only correctly mapped when they are contained in the last file (in this case b.js).

If I delete the "UUUUUU" part and move into into a.js instead, then the error messages I receive claim that the error occurs in b.js at line 3.

I should note that I am simply validating the files by consuming them through Chrome using Developer tools.

Regarding my code; it essentially looks exactly like yours above except for "sourceMap.EndFile(outputWriter, "\r\n");". If you still find a discrepancy between your results and what I claim then I can post the actual code.
Coordinator
Jun 16, 2013 at 9:55 AM
Yes, I'm seeing that too. Sorry; I should've tested this first in Chrome. So the command-line tool generates a map that works, but the DLL when sharing the same V3SymbolMap object and running MinifyJavaScript does not.
Okay, so I ran the command-line tool with the -reorder:no switch so the function declaration doesn't get moved to the very front of the combined.js:
ajaxmin a.js b.js c.js -o combined.js -map:v3 combined.js.map -reorder:no
Then I added the ReorderScopeDeclarartions=false flag to the CodeSettings object in my dll test app to match the command-line. (and went back to the proper a.js, b.js, c.js ordering of input files) The JS output by the two methods is identical:
define("jake");define("boo");console.log(123);UUUUUU;define("coo");function define(){}
/*
//@ sourceMappingURL=combined.js.map
*/
but the mapping is slightly different. The command-line mappings:
"mappings":"AAAAA,MAAM,CAAC,MAAD,CAAQ,CCAdA,MAAM,CAAC,KAAD,CAAO,CAEbC,OAAOC,IAAI,CAAC,GAAD,CAAK,CAEhBC,MACA,CCLAH,MAAM,CAAC,KAAD,CAAO,CAEbA,SAASA,MAAM,CAAA,CAAG",
and the dll mappings:
"mappings":"AAAAA,MAAM,CAAC,MAAD,CAAQ,dCAdA,MAAM,CAAC,KAAD,CAAO,CAEbC,OAAOC,IAAI,CAAC,GAAD,CAAK,CAEhBC,M,rCCJAH,MAAM,CAAC,KAAD,CAAO,CAEbA,SAASA,MAAM,CAAA,CAAG",
(map items #6, #18 and #19). Looking into why. And I'm assuming here that you've got a good work-around using the ///#SOURCE method, so this isn't a high priority.
Coordinator
Jun 16, 2013 at 10:21 AM
(ugh - typed a big long description and CodePlex failed to take it. At least I think it did. If there's a double-post, I apologize)

So I see the problem now. With the separate-MinifyJavaScript calls, the output visitor generating the map is being invoked separately for each file, and has no idea the output is being appended to previous output, so the "minified" code indexes start over at 1, which horks the file. I think I might be able to fix this, but I'm not really sure yet.

Good map file decode:
AAAAA   (0,0,0,0,0)     min:(1,1)       src: (1,1)       name:'define'          a.js
MAAM    (6,0,0,6)       min:(1,7)       src: (1,7)
CAAC    (1,0,0,1)       min:(1,8)       src: (1,8)
MAAD    (6,0,0,-1)      min:(1,14)      src: (1,7)
CAAQ    (1,0,0,8)       min:(1,15)      src: (1,15)
CCAdA   (1,1,0,-14,0)   min:(1,16)      src: (1,1)       name:'define'          b.js
MAAM    (6,0,0,6)       min:(1,22)      src: (1,7)
CAAC    (1,0,0,1)       min:(1,23)      src: (1,8)
KAAD    (5,0,0,-1)      min:(1,28)      src: (1,7)
CAAO    (1,0,0,7)       min:(1,29)      src: (1,14)
CAEbC   (1,0,2,-13,1)   min:(1,30)      src: (3,1)       name:'console'
OAAOC   (7,0,0,7,1)     min:(1,37)      src: (3,8)       name:'log'
IAAI    (4,0,0,4)       min:(1,41)      src: (3,12)
CAAC    (1,0,0,1)       min:(1,42)      src: (3,13)
GAAD    (3,0,0,-1)      min:(1,45)      src: (3,12)
CAAK    (1,0,0,5)       min:(1,46)      src: (3,17)
CAEhBC  (1,0,2,-16,1)   min:(1,47)      src: (5,1)       name:'UUUUUU'
MACA    (6,0,1,0)       min:(1,53)      src: (6,1)
CCLAH   (1,1,-5,0,-3)   min:(1,54)      src: (1,1)       name:'define'          c.js
MAAM    (6,0,0,6)       min:(1,60)      src: (1,7)
CAAC    (1,0,0,1)       min:(1,61)      src: (1,8)
KAAD    (5,0,0,-1)      min:(1,66)      src: (1,7)
CAAO    (1,0,0,7)       min:(1,67)      src: (1,14)
CAEbA   (1,0,2,-13,0)   min:(1,68)      src: (3,1)       name:'define'
SAASA   (9,0,0,9,0)     min:(1,77)      src: (3,10)      name:'define'
MAAM    (6,0,0,6)       min:(1,83)      src: (3,16)
CAAA    (1,0,0,0)       min:(1,84)      src: (3,16)
CAAG    (1,0,0,3)       min:(1,85)      src: (3,19)
Bad map file decode (notice the "min" values restarting at 1 for each file switch):
AAAAA   (0,0,0,0,0)     min:(1,1)       src: (1,1)       name:'define'          a.js
MAAM    (6,0,0,6)       min:(1,7)       src: (1,7)
CAAC    (1,0,0,1)       min:(1,8)       src: (1,8)
MAAD    (6,0,0,-1)      min:(1,14)      src: (1,7)
CAAQ    (1,0,0,8)       min:(1,15)      src: (1,15)
dCAdA   (-14,1,0,-14,0) min:(1,1)       src: (1,1)       name:'define'          b.js
MAAM    (6,0,0,6)       min:(1,7)       src: (1,7)
CAAC    (1,0,0,1)       min:(1,8)       src: (1,8)
KAAD    (5,0,0,-1)      min:(1,13)      src: (1,7)
CAAO    (1,0,0,7)       min:(1,14)      src: (1,14)
CAEbC   (1,0,2,-13,1)   min:(1,15)      src: (3,1)       name:'console'
OAAOC   (7,0,0,7,1)     min:(1,22)      src: (3,8)       name:'log'
IAAI    (4,0,0,4)       min:(1,26)      src: (3,12)
CAAC    (1,0,0,1)       min:(1,27)      src: (3,13)
GAAD    (3,0,0,-1)      min:(1,30)      src: (3,12)
CAAK    (1,0,0,5)       min:(1,31)      src: (3,17)
CAEhBC  (1,0,2,-16,1)   min:(1,32)      src: (5,1)       name:'UUUUUU'
M       (6)             min:(1,38)
rCCJAH  (-37,1,-4,0,-3) min:(1,1)       src: (1,1)       name:'define'          c.js
MAAM    (6,0,0,6)       min:(1,7)       src: (1,7)
CAAC    (1,0,0,1)       min:(1,8)       src: (1,8)
KAAD    (5,0,0,-1)      min:(1,13)      src: (1,7)
CAAO    (1,0,0,7)       min:(1,14)      src: (1,14)
CAEbA   (1,0,2,-13,0)   min:(1,15)      src: (3,1)       name:'define'
SAASA   (9,0,0,9,0)     min:(1,24)      src: (3,10)      name:'define'
MAAM    (6,0,0,6)       min:(1,30)      src: (3,16)
CAAA    (1,0,0,0)       min:(1,31)      src: (3,16)
CAAG    (1,0,0,3)       min:(1,32)      src: (3,19)
Jun 16, 2013 at 1:31 PM
Yes, I have everything working fine using "///#SOURCE" in my project. Making it work by reusing the V3SourceMap object should just be for a cleaner interface and workflow (maybe the sport of it). Definitely not a high priority.

Your identification of the index being reset also kind of makes sense. Looking at the incorrect result though, the pattern actually seems to be correct. As far as I can see the only problem is that the offset is not relative to where the previous index ended.

If you want I can try and have a go at it?
Coordinator
Jun 17, 2013 at 4:54 PM
I believe I have this fixed in the 4.94 release now. When I process a.js, b.js, and c.js with separate calls to MinifyJavaScript as described above, the generated source map ends up with these mappings:
AAAAA   (0,0,0,0,0)     min:(1,1)       src: (1,1)       name:'define'          a.js
MAAM    (6,0,0,6)       min:(1,7)       src: (1,7)
CAAC    (1,0,0,1)       min:(1,8)       src: (1,8)
MAAD    (6,0,0,-1)      min:(1,14)      src: (1,7)
CAAQ    (1,0,0,8)       min:(1,15)      src: (1,15)
CCAdA   (1,1,0,-14,0)   min:(1,16)      src: (1,1)       name:'define'          b.js
MAAM    (6,0,0,6)       min:(1,22)      src: (1,7)
CAAC    (1,0,0,1)       min:(1,23)      src: (1,8)
KAAD    (5,0,0,-1)      min:(1,28)      src: (1,7)
CAAO    (1,0,0,7)       min:(1,29)      src: (1,14)
CAEbC   (1,0,2,-13,1)   min:(1,30)      src: (3,1)       name:'console'
OAAOC   (7,0,0,7,1)     min:(1,37)      src: (3,8)       name:'log'
IAAI    (4,0,0,4)       min:(1,41)      src: (3,12)
CAAC    (1,0,0,1)       min:(1,42)      src: (3,13)
GAAD    (3,0,0,-1)      min:(1,45)      src: (3,12)
CAAK    (1,0,0,5)       min:(1,46)      src: (3,17)
CAEhBC  (1,0,2,-16,1)   min:(1,47)      src: (5,1)       name:'UUUUUU'
M       (6)             min:(1,53)
CCJAH   (1,1,-4,0,-3)   min:(1,54)      src: (1,1)       name:'define'          c.js
MAAM    (6,0,0,6)       min:(1,60)      src: (1,7)
CAAC    (1,0,0,1)       min:(1,61)      src: (1,8)
KAAD    (5,0,0,-1)      min:(1,66)      src: (1,7)
CAAO    (1,0,0,7)       min:(1,67)      src: (1,14)
CAEbA   (1,0,2,-13,0)   min:(1,68)      src: (3,1)       name:'define'
SAASA   (9,0,0,9,0)     min:(1,77)      src: (3,10)      name:'define'
MAAM    (6,0,0,6)       min:(1,83)      src: (3,16)
CAAA    (1,0,0,0)       min:(1,84)      src: (3,16)
CAAG    (1,0,0,3)       min:(1,85)      src: (3,19)
And gives the proper decoded location for the UUUUUU error (b.js, line 5).