Filtering blocks of code

[MilesCranmer]

I have some large test suites for some libraries – DynamicExpressions.jl and SymbolicRegression.jl – that I want to start rewriting in Literate.jl format to serve as an extended documentation. I like how Interfaces.jl does this in this file and wanted to try it with my own test suites.

However, these test suites are very large, and include a lot of not-useful cases that the user wouldn't need to see. Basically there is only 5% of these scripts that I would want Literate to render. So I was wondering what the best way is to filter all code except a specific block? As far as I can tell, the #src command can filter individual lines. So, I am wondering if there is something like #literate_begin and #literate_end that cause the parser to filter out that specific block of code?

I'm happy to just write a preprocessing step in my docs generation but basically I wanted to see if I was missing a common workflow that people use already.

[MilesCranmer]

Okay here's my current approach. This file is an example: https://github.com/SymbolicML/DynamicExpressions.jl/blob/0d1c19243753eb3ac1cbbf15527863c56d465a99/test/test_base_2.jl

The syntax is

    #literate_begin file="src/examples/base_operations.md"
    
    # ...
    
    #literate_end

I have this file: https://github.com/SymbolicML/DynamicExpressions.jl/blob/0d1c19243753eb3ac1cbbf15527863c56d465a99/docs/utils.jl

When you call process_literate_blocks(), it goes every file in the test/ folder. Whenever it matches a literate_begin block, it will extract all code, dedent it (based on min indentation in the block), and then generate some standalone julia files to pass to Literate.jl.

I think it works nicely. In principle this would also let you have Literate blocks within source code too, like

function my_monolithic_function(x, y)
    #= do stuff =#
    
    # Explain block of code:
    #literate_begin file="src/examples/how_it_works.md"
    
    # <Explain the most complex parts of the file>
    
    #literate_end
end

Thoughts @fredrikekre?

[fredrikekre]

This file is an example

But in this file everything except the first and last line are literated, so those two could just have #src, right?

I don't really have anything to add -- seems like there isn't really any changes needed in Literate? Perhaps this workflow can be mentioned in the Tips and Tricks section of the docs.

[MilesCranmer]

Good point. The file https://github.com/SymbolicML/DynamicExpressions.jl/blob/master/test/test_expressions.jl is a better example. You can see only the block at the end is used for the docs, while all the rest is ignored.

[MilesCranmer]

I was wondering if perhaps Literate.jl could support this syntax itself? I can make a PR with my parser and perhaps suggest a Literate API to use it. I think it is broadly useful so wouldn't want to keep it for myself. And also a standard syntax for this may be better than each person rolling their own parser. Wdyt?