Required files not found during build (e.g., noun_Key_1785639_crop.svg, noun_Email_3027864_ltgreen_svg-tex.pdf).

[kfogel]

On the jinjification branch (which is soon to be merged to master), some recent letterhead changes (probably commit cb9f802) made it so that the first time you build a doc you may get an error about file noun_Key_1785639_crop.svg not being findable. Actually, the error will say something like noun_Key_1785639_crop_svg-tex.pdf or something like that, because it's an intermediate file; unfortunately, I can't reproduce the error anymore ever since I successfully built by putting noun_Key_1785639_crop.svg in the current directory. Still, we should figure out what it was and fix it.

(Update: I figured out how to reproduce the error, and also why it intermittently wasn't reproducing. See subsequent comments.)

[kfogel]

Okay, this is happening now with another file, in two different builds, and one of those builds I know was working for James (because he built the doc and shipped it days ago). Here's that error:

$ make ehdp-open-path-development.pdf

...

No file ehdp-open-path-development.bbl.

Package svg Warning: You didn't enable `shell escape' (or `write18')
(svg)                so it wasn't possible to launch the Inkscape export
(svg)                for `noun_Email_3027864_ltgreen.svg' on input line 80.


! Package svg Error: File `noun_Email_3027864_ltgreen_svg-tex.pdf' is missing.

See the svg package documentation for explanation.
Type  H <return>  for immediate help.
 ...                                              
                                                  
l.80     }}
           
!  ==> Fatal error occurred, no output PDF file produced!
Transcript written on ehdp-open-path-development.log.
=== TeX engine is 'pdfTeX'
Latexmk: Non-existent bbl file 'ehdp-open-path-development.bbl'
 No file ehdp-open-path-development.bbl.
Latexmk: Errors, so I did not complete making targets
Collected error summary (may duplicate other messages):
  pdflatex: Command for 'pdflatex' gave return code 1
      Refer to 'ehdp-open-path-development.log' for details
Latexmk: Use the -f option to force complete processing,
 unless error was exceeding maximum runs, or warnings treated as errors.
make[1]: *** [/home/kfogel/private/work/ots/r/ots-doctools/Makefile:84: ehdp-open-path-development.pdf] Error 12
make: *** [Makefile:23: ehdp-open-path-development.pdf] Error 2

Clearly this is about latex/noun_Email_3027864_ltgreen.svg in the ots-doctools source tree, which at some point is supposed to get converted in a PDF.

Copying the .svg file into the current working directory where I'm building the doc doesn't solve the problem. And I remember that James and I encountered an error message like this one, mentioning "shell escape" and "write18", earlier, and that James solved it recently. Looking into this now.

[kfogel]

Oh very interesting:

If I revert commit aaf0281 (i.e., if I add back the --shell-escape flag that we removed because we were worried that it's a security hole), then the error changes. Now it says that the export using Inkscape failed, and "Please check in the log file how the invocation of Inkscape took place and try to execute it yourself in the terminal on input line 80.":

No file ehdp-open-path-development.bbl.

** (inkscape:284298): WARNING **: 11:07:58.715: Can't open file: noun_Email_3027864_ltgreen.svg (doesn't exist)

** (inkscape:284298): WARNING **: 11:07:58.721: Can't open file: noun_Email_3027864_ltgreen.svg (doesn't exist)

** (inkscape:284298): WARNING **: 11:07:58.721: Specified document noun_Email_3027864_ltgreen.svg cannot be opened (does not exist or not a valid SVG file)
system returned with code 256

Package svg Warning: The export with Inkscape failed for file
(svg)                `noun_Email_3027864_ltgreen.svg'
(svg)                Troubleshooting: Please check in the log file how
(svg)                the invocation of Inkscape took place and try to
(svg)                execute it yourself in the terminal on input line 80.


! Package svg Error: File `noun_Email_3027864_ltgreen_svg-tex.pdf' is missing.

See the svg package documentation for explanation.
Type  H <return>  for immediate help.
 ...                                              
                                                  
l.80     }}
           
!  ==> Fatal error occurred, no output PDF file produced!
Transcript written on ehdp-open-path-development.log.
=== TeX engine is 'pdfTeX'
Latexmk: Non-existent bbl file 'ehdp-open-path-development.bbl'
 No file ehdp-open-path-development.bbl.
Latexmk: Errors, so I did not complete making targets
Collected error summary (may duplicate other messages):
  pdflatex: Command for 'pdflatex' gave return code 1
      Refer to 'ehdp-open-path-development.log' for details
Latexmk: Use the -f option to force complete processing,
 unless error was exceeding maximum runs, or warnings treated as errors.
make[1]: *** [/home/kfogel/private/work/ots/r/ots-doctools/Makefile:84: ehdp-open-path-development.pdf] Error 12
make: *** [Makefile:23: ehdp-open-path-development.pdf] Error 2

Here's what the relevant portion of ehdp-open-path-development.log says:

runsystem(inkscape -z -D --export-latex  --file="noun_Email_3027864_ltgreen.svg" --export-pdf="noun_Email_3027864_ltgreen_svg-tex.pdf" )...executed.

I'll bet if I copy latex/noun_Email_3027864_ltgreen.svg from ots-doctools into my source tree now (note that it wasn't there for the above run), the build would work. Let me try that... Yup! (Well, I had to also copy in noun_Key_1785639_ltgreen.svg and noun_Telephone_2591756_ltgreen.svg, but you get the idea -- the hypothesis is still confirmed.)

Next, let's test the other part of the hypothesis:

If I leave those files in place in the build directory, make clean, remove the generated svn-inkscape/ directory (because it's caching noun_Email_3027864_ltgreen_svg-tex.pdf and friends, meaning the build process would never try to invoke Inkscape in the first place), remove the --shell-escape local mod in the ots-doctools Makefile, and try again, it should fail with the original error. And it does!

So, there are really two bugs here:

1. In our normal modern configuration (without --shell-escape in the Makefile), we fail to invoke Inkscape at all, so those generated PDF files of our nouns never get built.

2. Even if we do invoke Inkscape (say, by temporarily adding back --shell-escape), the build still fails because Inkscape is looking for the source .svg files in the current build directory, and they're not there -- they're over in the ots-doctools/latex/ directory.

Also, our make clean process probably should remove the svg-inkscape/ subdirectory from the build directory, because it was the Makefile that generated that subdirectory in the first place. However, it's kind of convenient for the moment that make clean doesn't remove it, because preservation of that subdirectory is part of the workaround for problems (1) and (2) above. Once we've fixed them, we should update the clean rule as well.

[kfogel]

A thought: as part of the fix, we could pre-build the needed .pdf files. That is, just version them in the ots-doctools tree, even though they'd be generated from the .svg files already versioned there. While versioning generated files is not a great thing, it might be a reasonable solution here. It would be one way to avoid adding back --shell-escape (in the Makefile's invocation of pdflatex via latexmk).

[kfogel]

The following script can be used to temporarily fix builds. Note it passes -shell-escape instead of --shell-escape to pdflatex. Apparently the program accepts both the one-hyphen and the two-hyphen version of the long option.

#!/bin/sh

cp ${OTS_DOCTOOLS_DIR}/latex/noun_Email_3027864_ltgreen.svg .

cp ${OTS_DOCTOOLS_DIR}/latex/noun_Telephone_2591756_ltgreen.svg .

sed -e 's/-halt-on-error/-halt-on-error -shell-escape/' \
    < ${OTS_DOCTOOLS_DIR}/Makefile > ${OTS_DOCTOOLS_DIR}/tmp-$$-Makefile
mv ${OTS_DOCTOOLS_DIR}/tmp-$$-Makefile ${OTS_DOCTOOLS_DIR}/Makefile

[jvasile]

We used to have write18 enabled (that's what shell-escape does) but you were of the opinion that it was a security issue, so we removed it since we didn't think we needed it. Turns out it was doing something useful after all.

I do think your security concerns make sense. The right answer here is to do the svg->pdf conversion in a separate process (perhaps using image magick instead of inkscape) and then make that pdf available to latex. That might mean changing the source to look include pdf images instead of svg. How to do all this I haven't yet solved, but I propose that as the general shape of a solution.

[kfogel]

Well, adding back the shell-escape solves part of the problem (as this comment notes), but there's still some problem left after that.

I mean, we'd still have the security concern anyway, and I agree that it should be taken seriously and that doing the conversion in a separate process is the solution. But there's a bug here independently of that, according to my tests earlier at least.

[kfogel]

Closing in favor of issue #15.