Fibonacci calculation at peak performance

pxp9 — Sat, 20 Sep 2025 00:00:00 +0000

Idea</h2>
The idea of this article is to explore the Fibonacci sequence algorithms in Elixir, explore optimizations which can be done and explore how native code can help us in order to compute even faster. The main objective of the article is to compute the largest Fibonacci number as fast as possible.

Start with the Math !</h3>
Most powerful algorithms out there rely on a quite simple math principle.
$$ Fn = F_{n-1} + F_{n-2} $$
This principle means that if you know the 2 previous terms of the series you can compute the next one. So the Fibonacci series is recursive because in the definition of the series appears the series itself.

Math is not real life.</h3>
Based on the math we have seen before we can think this algorithm:
 def fib(0), do: 0 def fib(1), do: 1 def fib(n) do fib(n - 1) + fib(n - 2) end</code></pre> Let me tell you why this is a terrible idea. This algorithm is super slow, because it does extra computations which are already computed and it does not store any result.
flowchart TB; root["$$F_{n}$$"] f1["$$F_{n-1}$$"] f2["$$F_{n-2}$$"] f11["$$F_{n-2}$$"] f12["$$F_{n-3}$$"] f21["$$F_{n-3}$$"] f22["$$F_{n-4}$$"] root-->f1; root-->f2; f1-->f11; f1-->f12; f2-->f21; f2-->f22; </pre>
As you can see from this example, the complexity of this algorithm grows exponentially $O(2^n)$. Notice that we are doing multiple times the same computation $F_{n-2}$ or $F_{n-3}$.
Improving naive solution.</h3>
One improvement we can do to the solution is to apply Dynamic programming</a>. We can optimize the algorithm by doing memoization, so the complexity can be reduced a lot.
def fib(n) do if :ets.whereis(:fibs) == :undefined do :ets.new(:fibs, [:named_table, read_concurrency: true]) end do_fib(n) end defp do_fib(0) do :ets.insert(:fibs, {0, 0}) 0 end defp do_fib(1) do :ets.insert(:fibs, {1, 1}) 1 end defp do_fib(n) do result = :ets.lookup(:fibs, n) if result == [] do val = do_fib(n - 1) + do_fib(n - 2) :ets.insert(:fibs, {n, val}) val else [{_n, val}] = result val end end </code></pre>
This is the same algorithm as before, but we cache the results in an ETS table</a>, so if we have an already computed result, we do not need to recompute it all.
We can do a simple benchmark between this implementation with memoization and the other one with not, so it demonstrates how powerful is this technique.
##### With input 1 ##### Name ips average deviation median 99th % Elixir O(2^n) Algorithm 16.64 M 60.09 ns ±42065.44% 45 ns 69 ns Elixir O(2^n) Algorithm Memo 1.16 M 864.46 ns ±5353.57% 792 ns 1150 ns Comparison: Elixir O(2^n) Algorithm 16.64 M Elixir O(2^n) Algorithm Memo 1.16 M - 14.39x slower +804.37 ns ##### With input 20 ##### Name ips average deviation median 99th % Elixir O(2^n) Algorithm Memo 1.51 M 0.66 μs ±9240.96% 0.54 μs 0.88 μs Elixir O(2^n) Algorithm 0.0102 M 98.09 μs ±55.81% 96.94 μs 105.33 μs Comparison: Elixir O(2^n) Algorithm Memo 1.51 M Elixir O(2^n) Algorithm 0.0102 M - 148.28x slower +97.43 μs ##### With input 45 ##### Name ips average deviation median 99th % Elixir O(2^n) Algorithm Memo 1.48 M 0.00000 s ±8725.95% 0.00000 s 0.00000 s Elixir O(2^n) Algorithm 0.00000 M 16.21 s ±0.13% 16.21 s 16.22 s Comparison: Elixir O(2^n) Algorithm Memo 1.48 M Elixir O(2^n) Algorithm 0.00000 M - 24065479.81x slower +16.21 s</code></pre>
--- config: themeVariables: xyChart: plotColorPalette: '#D278AA, #7C6D91' --- xychart title "Benchmark Memo vs No Memo" x-axis "Algorithm" [ "O(2^N) input 45", "Memo input 45", "O(2^N) input 20", "Memo input 20" ] y-axis "Time in microseconds" 1 --> 200 bar [1622000, -1000000, 105.33, -1000000 ] bar [-1000000, 1.150, -1000000, 0.88] </pre>
For each input we flush the memoization cache, because it will be unfair if we leave it, since it will not compute every N at least once.
Sometimes improving is not enough.</h3>
Dynamic programming implementation is pretty fast, but it is not enough. We can get a better implementation if we rethink the naive approach to make it $O(n)$.
One trick we can do is to store the previous 2 numbers as parameters of the recursion so this way, we dont need to compute all the other results each time.
def fib(0), do: 0 def fib(1), do: 1 def fib(n), do: fib(n, 0, 1) defp fib(1, _a, b), do: b defp fib(n, a, b), do: fib(n - 1, b, a + b)</code></pre> This implementation still uses the same mathematical principle, but it does less iterations, and it uses way more less storage since it just needs to store the 2 previous fibonacci numbers.
flowchart LR; root["$$F_{n}$$"] f1["$$F_{n-1}$$"] f2["$$F_{n-2}$$"] f0["$$F_{0}$$"] f01["$$F_{1}$$"] root-->f1; f1-->f2; f2-->f01; f01-->f0; </pre>
What about another math approach ?</h3>
There is another mathematical formula called Binet's formula</a>.
$$ Fn = \frac{\phi^{n} - (-\phi^{-n})} {\sqrt{5}} $$
This principle allow us to calculate any number of the sequence by just knowing the value of $\phi$ constant.
def phi_formula(n), do: round((:math.pow(@phi, n) - :math.pow(-@phi, -n)) fast_inverse_square_root(5.0)) defp fast_inverse_square_root(number) do x2 = number 0.5 <<i::integer-size(64)>> = <<number::float-size(64)>> <<y::float-size(64)>> = <<0x5FE6EB50C7B537A9 - (i >>> 1)::integer-size(64)>> y = y (1.5 - x2 y y) y = y (1.5 - x2 y y) y = y (1.5 - x2 y y) y end </code></pre>
This implementation is in fact $O(1)$ or $O(log n)$, depends on the complexity of `:math.pow</code> function. What is the tradeoff ? This implementation is an approximation so you will be stacking error. Notice that one divided the square root of 5 is calculated with the` `quake 3 algorithm</a>, another alternative would be to hardcode this constant in the algorithm.`
The end ?</h3> So... Is this the end ? NO, because you should know that Elixir runs in the Erlang VM which probably has some overhead while calculating. We can try to make an Erlang NIF</code> (Native Implementation Function), to see if it is faster than just standard Elixir. To do so, we can take a look to rustler</code>, which is fantastic crate</code>/elixir library. This is the code we will be using in Rust, #[rustler::nif] fn fib(n: u128) -> u128 { let mut a = 0; let mut b = 1; let mut i = 1; while i < n { b = a + b; a = b - a; i += 1; } b }</code></pre> and for the $\phi$ based algorithm, const PHI: f64 = 1.618033988749895; fn inv_sqrt(x: f64) -> f64 { let i = x.to_bits(); let x2 = x 0.5; let mut y = f64::from_bits(0x5FE6EB50C7B537A9 - (i >> 1)); y = y (1.5 - x2 y y); y = y (1.5 - x2 y y); y (1.5 - x2 y y) } #[rustler::nif] fn phi_formula(n: i32) -> f64 { ((PHI.powi(n) - (-PHI).powi(-n)) inv_sqrt(5.0)).ceil() }</code></pre> Of course the phi_formula</code> implementation has the same issue as the elixir one since it is an approximation, it should not be super precise at large scale. Lets talk about the Rust implementation of Fibonacci, as you can see is $O(N)$ and notice we are just using 2 variables instead of 3, this is because an arithmetic operation should be faster than storing in a CPU register. Also notice that Rust forces us to put types and, we are using u128</code>, so this way we can compute and store really big integer. Wait something weird is happening</h3> When I was testing the code for this article, I realized that sometimes Rust implementation was not consistant with the Elixir one, especially at large scale. ## Elixir impl with 100 input iex(1)> FibRustElixir.fib(100) 354224848179261915075 ## Rust impl with 100 input iex(2)> TurboFibonacci.fib(100) 354224848179261915075 ## Elixir impl with 200 input iex(3)> FibRustElixir.fib(200) 280571172992510140037611932413038677189525 ## Rust impl with 200 input iex(4)> TurboFibonacci.fib(200) 178502649656846143791255889261670949781</code></pre> It seems everything is correct, for 100</code>, but things are getting weird for 200</code>. What the hell the same algorithm gives different result ? The answer is called overflow. Since Rust is working with limited size integers it overflows for big numbers. Why Elixir does not have this problem ? It is because Elixir integers are dynamically sized. It creates a little bit of overhead because you must do allocations to fit the integer, but it allows you to have an "infinite" integer. I recommend this read Learning Elixir: Understanding Numbers</a>, in order to have a better understanding of what Elixir is doing. Then I wondered, What if I can make the same Elixir behaviour in Rust ? Thankfully, there are already super smart people in the Rust ecosystem that thought about this problem, and I observed that there was already a crate</code> for this num_bigint</code>. Another thing, one of the guys who did rustler</code>, thought about this problem and he integrated rustler</code> with num_bigint</code>, so this way you can have a translation layer with these kind of integers in Rust. Take a look the Encoder impl for BigInt</a>, this is the translation layer for Rust BigInt</code> to Elixir term</code> which is an integer</code>. Lets take a look to the implementation with BigInt</code> #[rustler::nif] fn fib_bignums(n: u128) -> BigInt { let mut a = BigUint::zero(); let mut b = BigUint::one(); let mut i = 1; while i < n { b = &a + &b; a = &b - &a; i += 1; } BigInt::from_biguint(Sign::Plus, b) }</code></pre> As you can see nothing really changes, now we are using the big integer which are resizable, and allow us to store massive integers. Lets try with this implementation and see if it gives the same result as Elixir one. ## Elixir with input 200 iex(3)> FibRustElixir.fib(200) 280571172992510140037611932413038677189525 ## Rust `u128` with input 200 iex(4)> TurboFibonacci.fib(200) 178502649656846143791255889261670949781 ## Rust resizable integers with input 200 iex(5)> TurboFibonacci.fib_bignums(200) 280571172992510140037611932413038677189525</code></pre> YES! It works! Now we can compute and store massive Fibonacci calculations. Theory pretty cool, but show me the numbers.</h3> Ok ok, now is the moment you were waiting for! The benchmarks and see if the theory is confirmed. The benchmark library I will be using is Benchee</a>, this library allow me to compare and obtain stats for each implementation and different inputs. I would like to explain I have divided the benchmarking in 2. Relative small inputs, less than 100</code>, and really big inputs from 200_000</code>. These are the specs of my system Operating System: Linux CPU Information: 13th Gen Intel(R) Core(TM) i5-1335U Number of Available Cores: 12 Available memory: 15.30 GB Elixir 1.18.4 Erlang 27.3.4.3 JIT enabled: true</code></pre> This is the setup for low inputs. def bench() do Benchee.run( %{ "Elixir O(N) Algorithm" => &fib/1, "Elixir O(2^n) Algorithm Memo" => &dp_slow_fib/1, "Elixir O(1) Algorithm Phi formula" => &phi_formula/1, "Rust O(1) Algorithm Phi formula" => &TurboFibonacci.phi_formula/1, "Rust O(N) Algorithm" => &TurboFibonacci.fib/1, "Rust O(N) Algorithm expanding nums" => &TurboFibonacci.fib_bignums/1 }, inputs: %{ "1" => 1, "71" => 71, "100" => 100 }, parallel: 2, after_scenario: fn _input -> if :ets.whereis(:fibs) != :undefined do :ets.delete(:fibs) end end ) nil end</code></pre> Before you look the results you need to consider that Rust expanding nums implementation was run with the dirty scheduler in Erlang, without it is still the worst, because of the overhead it has, but not so much worse around 33x slower. Also consider invalid $\Phi$ formula implementations for the input >= 71</code> or input >= 75</code>, because they start to not give a precise value after that. As you can see here the result for input 71 is correct for 72 is not, as well in the $\phi$ elixir implementation. ### Input 71 iex(2)> TurboFibonacci.phi_formula(71) 308061521170129.0 iex(3)> TurboFibonacci.fib(71) 308061521170129 ### Input 72 iex(4)> TurboFibonacci.phi_formula(72) 498454011879263.0 iex(5)> FibRustElixir.phi_formula(72) 498454011879264 iex(6)> TurboFibonacci.fib(72) 498454011879264 ### Input 75 iex(12)> FibRustElixir.phi_formula(75) 2111485077978050 iex(13)> TurboFibonacci.fib(75) 2111485077978050 ### Input 76 iex(14)> FibRustElixir.phi_formula(76) 3416454622906708 iex(15)> TurboFibonacci.fib(76) 3416454622906707</code></pre> Here are the results for the benchmark low inputs. ##### With input 1 ##### Name ips average deviation median 99th % Elixir O(N) Algorithm 16.38 M 61.07 ns ±45774.27% 44 ns 71 ns Rust O(1) Algorithm Phi formula 9.15 M 109.26 ns ±4221.28% 100 ns 210 ns Rust O(N) Algorithm 8.26 M 121.09 ns ±23343.76% 100 ns 203 ns Elixir O(1) Algorithm Phi formula 3.31 M 302.31 ns ±136.52% 260 ns 531 ns Elixir O(2^n) Algorithm Memo 1.19 M 838.18 ns ±5624.56% 787 ns 1101 ns Rust O(N) Algorithm expanding nums 0.188 M 5308.94 ns ±84.14% 4395 ns 16176.06 ns Comparison: Elixir O(N) Algorithm 16.38 M Rust O(1) Algorithm Phi formula 9.15 M - 1.79x slower +48.19 ns Rust O(N) Algorithm 8.26 M - 1.98x slower +60.03 ns Elixir O(1) Algorithm Phi formula 3.31 M - 4.95x slower +241.25 ns Elixir O(2^n) Algorithm Memo 1.19 M - 13.73x slower +777.11 ns Rust O(N) Algorithm expanding nums 0.188 M - 86.93x slower +5247.87 ns ##### With input 100 ##### Name ips average deviation median 99th % Rust O(1) Algorithm Phi formula 8.09 M 123.63 ns ±263.70% 113 ns 195 ns Rust O(N) Algorithm 4.63 M 215.80 ns ±356.51% 198 ns 419 ns Elixir O(1) Algorithm Phi formula 3.15 M 317.82 ns ±209.58% 277 ns 615 ns Elixir O(2^n) Algorithm Memo 1.45 M 687.71 ns ±8165.87% 587 ns 924 ns Elixir O(N) Algorithm 0.66 M 1517.15 ns ±2172.62% 1380 ns 1928 ns Rust O(N) Algorithm expanding nums 0.0784 M 12753.08 ns ±55.31% 13905 ns 24389 ns Comparison: Rust O(1) Algorithm Phi formula 8.09 M Rust O(N) Algorithm 4.63 M - 1.75x slower +92.17 ns Elixir O(1) Algorithm Phi formula 3.15 M - 2.57x slower +194.18 ns Elixir O(2^n) Algorithm Memo 1.45 M - 5.56x slower +564.08 ns Elixir O(N) Algorithm 0.66 M - 12.27x slower +1393.51 ns Rust O(N) Algorithm expanding nums 0.0784 M - 103.15x slower +12629.45 ns ##### With input 71 ##### Name ips average deviation median 99th % Rust O(1) Algorithm Phi formula 7.75 M 128.97 ns ±4535.63% 117 ns 230 ns Rust O(N) Algorithm 6.54 M 152.88 ns ±19215.97% 123 ns 224 ns Elixir O(1) Algorithm Phi formula 3.40 M 294.01 ns ±339.67% 258 ns 533 ns Elixir O(2^n) Algorithm Memo 1.43 M 700.42 ns ±9294.91% 581 ns 904 ns Elixir O(N) Algorithm 1.15 M 869.80 ns ±184.80% 853 ns 1259 ns Rust O(N) Algorithm expanding nums 0.0940 M 10633.12 ns ±73.98% 10948 ns 20597 ns Comparison: Rust O(1) Algorithm Phi formula 7.75 M Rust O(N) Algorithm 6.54 M - 1.19x slower +23.91 ns Elixir O(1) Algorithm Phi formula 3.40 M - 2.28x slower +165.04 ns Elixir O(2^n) Algorithm Memo 1.43 M - 5.43x slower +571.45 ns Elixir O(N) Algorithm 1.15 M - 6.74x slower +740.83 ns Rust O(N) Algorithm expanding nums 0.0940 M - 82.44x slower +10504.15 ns</code></pre> --- config: themeVariables: xyChart: plotColorPalette: '#B7410E, #7C6D91' --- xychart title "Low numbers benchmark input 1" x-axis "Algorithm" ["Rust Phi", "Elixir Phi", "Rust O(N)", "Elixir O(N)", "Elixir O(2^N) Memo" ] y-axis "Time in nanoseconds" 1 --> 1200 bar [210, -1000, 203 ,-1000, -1000] bar [-1000, 531, -1000 , 71, 1101] </pre> --- config: themeVariables: xyChart: plotColorPalette: '#B7410E, #7C6D91' --- xychart title "Low numbers benchmark input 71" x-axis "Algorithm" ["Rust Phi", "Elixir Phi", "Rust O(N)", "Elixir O(N)", "Elixir O(2^N) Memo" ] y-axis "Time in nanoseconds" 1 --> 1300 bar [230, -1000, 224 ,-1000, -1000] bar [-1000, 533, -1000 , 1259, 904] </pre> --- config: themeVariables: xyChart: plotColorPalette: '#B7410E, #7C6D91' --- xychart title "Low numbers benchmark input 100" x-axis "Algorithm" ["Rust O(N)", "Elixir O(N)", "Elixir O(2^N) Memo" ] y-axis "Time in nanoseconds" 1 --> 2000 bar [419,-1000, -1000] bar [-1000 , 1928, 924] </pre> The conclusion we can make here is that for really small inputs Rust NIF is worth it to use even if Elixir wins for input 1 because it is a clause in the function almost without any overhead. $\Phi$ algorithm is not so fast for low inputs so it is not worth to use. Notice that Rust expandable integers implementation has a super big overhead which is not compensate with the small input. This is the setup I was doing for Big numbers, As you can see here we have much bigger inputs, other functions cannot be included in this benchmark because they will lose precision</code>, they will take too much time</code> or they will take too much memory</code>. def bench_large_numbers() do Benchee.run( %{ "Elixir O(N) Algorithm" => &fib/1, "Rust O(N) Algorithm expanding nums" => &TurboFibonacci.fib_bignums/1 }, inputs: %{ "200_000" => 200_000, "500_000" => 500_000, "1 Million" => 1_000_000, "2 Million" => 2_000_000, }, parallel: 2 ) end</code></pre> These are the results for the big numbers benchmark. ##### With input 1 Million ##### Name ips average deviation median 99th % Rust O(N) Algorithm expanding nums 0.0890 11.23 s ±0.09% 11.23 s 11.24 s Elixir O(N) Algorithm 0.0798 12.54 s ±0.00% 12.54 s 12.54 s Comparison: Rust O(N) Algorithm expanding nums 0.0890 Elixir O(N) Algorithm 0.0798 - 1.12x slower +1.30 s ##### With input 2 Million ##### Name ips average deviation median 99th % Elixir O(N) Algorithm 0.0238 0.70 min ±2.11% 0.70 min 0.71 min Rust O(N) Algorithm expanding nums 0.0101 1.64 min ±0.01% 1.64 min 1.64 min Comparison: Elixir O(N) Algorithm 0.0238 Rust O(N) Algorithm expanding nums 0.0101 - 2.35x slower +0.94 min ##### With input 200_000 ##### Name ips average deviation median 99th % Rust O(N) Algorithm expanding nums 3.04 328.71 ms ±0.56% 328.21 ms 333.74 ms Elixir O(N) Algorithm 1.50 668.81 ms ±5.29% 659.64 ms 728.12 ms Comparison: Rust O(N) Algorithm expanding nums 3.04 Elixir O(N) Algorithm 1.50 - 2.03x slower +340.10 ms ##### With input 500_000 ##### Name ips average deviation median 99th % Rust O(N) Algorithm expanding nums 0.44 2.25 s ±0.14% 2.25 s 2.25 s Elixir O(N) Algorithm 0.23 4.33 s ±2.59% 4.33 s 4.43 s Comparison: Rust O(N) Algorithm expanding nums 0.44 Elixir O(N) Algorithm 0.23 - 1.93x slower +2.09 s</code></pre> --- config: themeVariables: xyChart: plotColorPalette: '#B7410E, #7C6D91' --- xychart title "Big numbers benchmark" x-axis "Algorithm and input" ["E(200k)", "R(200k)", "E(500k)", "R(500k)", "E(1M)", "R(1M)" ] y-axis "Time in seconds" 0 --> 15 bar [-1, 0.33374, -1 , 2.25, -1, 11.24] bar [0.72812, -1, 4.43 ,-1, 12.54, -1] </pre> --- config: themeVariables: xyChart: plotColorPalette: '#B7410E, #7C6D91' --- xychart title "Big numbers benchmark (2 million)" x-axis "Algorithm and input" ["E(2M)", "R(2M)"] y-axis "Time in seconds" 1 --> 100 bar [ -1 , 98.4 ] bar [42.6 ,-1 ] </pre> This is really interesting result, since it seems Rust expanding nums it pays the overhead by winning Elixir to all the inputs except for 2 million, I believe why Elixir wins in 2 million input is because some sort of reallocation optimization which is not in Rust. Conclusion</h3> For every peak performance algorithm you should analyze your inputs and make correct assumptions like input <= 71</code>, so this way you benefit of the approximation algorithm. If your inputs are super large, consider using native code, instead of working in VM code. It seems to pay off the Native code, in fact Elixir is kinda cheating because Erlang is probably calling some C code to compute this kinda numbers. Benchmark! Always test your things with stress, load and speed benchmarking, this way you can ensure that your algorithm is gonna behave how you will expect. Of course there is much more strategies here which I did not cover, like Multi-threading, dividing the work in chunks and so on... Hopefully you liked the article, all the code of the article is available in Github</a>

Lazy Doc, Your documentation tool has arrived.

pxp9 — Tue, 04 Mar 2025 00:00:00 +0000

Idea</h2>
All started with a conversation with my boss, I was talking to him about a game we play (Balatro, wonderful game), and suddenly he asked, Are you using AI to program ? I told him that current the state of the AI is not that good for suggesting production code, but it is good to suggest other things like explanations, suggest syntax fixes, etc... One of the problems we have in my company codebase is that we have so much undocumented code, which is not a big problem at all, because we have a really clear API of how to interact, but each time someone new comes to the project, you need to explain the localized knowledge. We also talked about AI writing docs might be good, but we did not even know because we had not tested yet. That was how Lazy Doc idea came from.

What is Lazy Doc ?</h2>
Lazy Doc is a tool to improve your nonexistent or missing Elixir docs via AI. It detects undocumented functions, it gives the source code to the AI and it generates documentation based on it.
You just need to install it and configure it, execute `mix lazy_doc</code> and boom you have brand new documentation from scratch.`
`Purpose</h3> The purpose of Lazy Doc</code> is much bigger it aims to be a docs checker as well. with mix lazy_doc.check</code> you can check if all the documentation under patterns</code> paths regex. This check can be easily implemented in the CI, so if you forget to document something CI will not pass.`
`Documentation criteria</h3> Lazy Doc will only document functions which agree with the following criteria:`

Public functions not already documented. (already documented functions are ignored)</li>
Public functions without hidden documentation tag `@doc false</code>. (hidden docs function are ignored).</li>`
`Private functions are ignored (but included in the prompt if they are auxiliary functions continue reading for more info).</li> </ul> so this way if lazy_doc.check</code> does not pass because of a function, you take one of the following actions.``Make the function private</li> Make the doc hidden</li>`
Document or just execute mix lazy_doc</code>.</li> </ul> The main reason LazyDoc can be really good for your project, it is because it will enforce you to write documentation. Even if you do not want it, it offers a good start template. How is it built ?</h2> Okay, you want to build a an automatic documentor, right ? So the first step is to detect what the hell you should document and what you should not and leave it as it is. What alternatives do you have to detect undocumented functions ? You always need to read the file at least once and then you have multiple approaches. Process line by line approach</h3> This approach consist to check each line and look for search patterns, in our case @doc whatever</code> on top of a function. There are a bunch of problems with this approach: how do you get the code of a single function and its clauses ? you can try to read lines until one end , but how do you know if it is an end</code> of a case</code> or an end</code> of the def</code>.</li> how do you retain the format of the file ?</li> how do you detect inner inner modules ?</li> how do you detect if a function is inside an inner module or not ?</li> </ul> As you can see this is a simple approach, but it has a bunch of problems and probably many more problems, which I had not thought, because I did not even try to implement it this way. So what is the alternative ? ASTs for the king approach</h3> Okay, so we are gonna try to detect functions through the AST of the module but first, let's explain what an AST is. An AST is a data structure which contains the syntax of the code, in fact it stands for Abstract Syntax Tree. In Elixir an AST has always the same pattern. {:defmodule = name, [...] = meta, [...] = children } = node</code></pre> So you get the AST of file, via Code.string_to_quote()</code> function. For instance this module, it will have the corresponding AST defmodule Example do @doc "converts to string" def number(n) do Kernel.to_string(n) end end</code></pre> Note: you can experiment with ast.ninja site</a> if you want to experiment with ASTs {:defmodule, [line: 1], [ {:aliases, [line: 1], [:Example]}, [ do: {:block, [], [ {:@, [line: 2], [ {:doc, [line: 2], ["converts to string"]} ]}, {:def, [line: 3], [ {:number, [line: 3], [{:n, [line: 3], nil}]}, [ do: {{:., [line: 4], [ {:aliases, [line: 4], [:Kernel]}, :to_string ]}, [line: 4], [{:n, [line: 4], nil}]} ] ]} ]} ] ]}</code></pre> You can just pattern match the node you want and get the code from that node. for example: {:def = name, [...] = meta, [...] = children } = ast_fun</code></pre> In ast_fun</code> we have the sub ast of that function we just need to convert it back to code and send it to analyze with AI, but just if it is not documented. And you know that number</code> function is in the Example</code> module because it was in their children nodes list. How do you convert AST to String again ?</h4> You can just use Macro.to_string(ast_fun)</code>. (There is some caviar on this, continue reading). How do we know if a function is already documented or not ?</h4> Easy, you just need to call Code.fetch_docs(module)</code> which requires to give the name of the module. Not so fast, what happens if you have the same name of function with different number of arguments (different arities not clauses). This function will return you documentation for each different function in the language, but are 2 functions with the same name different from each other, just because their arity differs? The answer is YES, but does this make sense in the POV of documentation. Think for a moment of what coding situation would you use the same name with different arities. Let's take a look this example: ## Look this function, does make sense if you hide the other functions ? def fibs(n) do fibs(n, [1, 0]) end def fibs(1, [a, b | rest]), do: [a + b, a, b | rest] def fibs(n, [a, b | rest]) do fibs(n - 1, [a + b, a, b | rest]) end</code></pre> Probably just because one of the functions is auxiliary of the other. So the auxiliary function does not make sense by itself and the entry point function does not make sense without the auxiliary. We need a way to group up the functions by name even if they have different arity and clauses (function same name and same arity) to send it to AI in a single prompt. Fetch the prompt</h3> Okay, you need a way to get the documentation using AI So what you should do is to call an AI API, but which one ? There is a bunch of AI APIs and everyone has its favorite AI. Not talking about if you want to use an internal model of the company, it could be really hard. So why not give freedom to call your own AI ? You just need to implement the LazyDoc.Provider</code> behavior. This is the contract that will use LazyDoc</code> to request the docs to API and extract them from the response. Make a good prompt (AI is like a kid explain what you want).</h3> Making a good prompt is really important for fetching the docs, you should specify your prompt always with at least one example and be precise with the format you want. Example: ~s(You should describe the parameters based on the spec given and give a small description of the following function.\n\nPlease do it in the following format given as an example, important do not return the header of the function, do not return a explanation of the function, your output must be only the docs in the following format:\n\nReturns the Transaction corresponding to transaction_id$Initial sentence indicating what returns the function$\n\n## Parameters\n\n- transaction_id - foreign key of the Transactions table.\n## Description\n Performs a search in the database\n\nFunction to document:\n)</code></pre> This is the default prompt the library will use, to this prompt you should concatenate the function code. Of course there is a bunch of attempts to get the right prompt and you need to even consider to be compliant with existing doc tools. Insert the docs on the AST</h3> To insert the doc in the AST you just need to create a @doc</code> node and insert it just before def</code> node. but what if this def</code> node is in an inner module children. You need to find first the children of the corresponding module. JUST pattern match, it will work (complex pattern match but I promise, it will work). Once you have the children, need to locate the def</code> node position in the children and then insert it just one position before. The end of the trip ?</h3> And then transform the AST to String with Macro.to_string()</code> and write it to the corresponding file, right ? WRONG, you just messed up all the format of the original file. What? Where are my comments? it just deleted my comments during the entire process. How do you fix it ? The master talk about AST manipulation.</h3> At this point, you are almost there, almost done, but you are kind of stuck because you want to preserve the code as it was. Remember this ? leave the code as it is </blockquote> It is one of the premises that we have at least to use it in production. Surprise surprise, you have seen it before, it is the AST NINJA</code> talk. Actually the name of the talk is The Elixir parser under microscope</code> by Arjan Scherpenisse</code>. This talk is about how to preserve the code as it is after an AST manipulation. Basically what you need to do is what the Elixir Formatter does, Add tokenizer metadata to AST, store the comments outside the AST and merge it after all the AST manipulation. Fortunately this talk was done years ago and the Elixir std library changed according the talk recommendations, so we just need to use Code.string_to_quote_with_comments</code> with some options and then after the AST manipulation use ast |> quote_to_algebra(comments: comments, ...) |> Inspect.Algebra.format(max_line_width_configured) |> IO.iodata_to_binary()</code></pre> This will preserve the format of the file. Feature requests</h2> After telling you how you can build LazyDoc, I can tell you what features request were requested. They requested a way to not have bloated documentation files, the issue users found was that generating so much docs will cause the file incresize the size dramatically. So in order to address it, I added the external_doc</code> option which allows you to write the doc in a separate markdown file and read the doc of that function at compile time.</li> </ul> Example: priv/lazy_doc/lazy_doc/docs_per_module.md</code> Returns a list of documentation for each function in the specified modules. ## Parameters - modules - a list of module names to extract documentation from. ## Description Retrieves the documentation for functions in the given modules using Elixir's introspection capabilities.</code></pre> lib/lazy_doc.ex</code> @doc File.read!("priv/lazy_doc/lazy_doc/docs_per_module.md") def docs_per_module(modules) do ... end </code></pre> this way, you can enjoy a rich documentation without having big blocks of documentation before each function. Even the LSP will give you the documentation if you hover over the function, because the documentation is embedded in the .beam</code> file. In the article, we focus on the functions because they were more interesting to analyze how they were done, but the library can also generate @moduledoc</code> based on the code of the module, but usually a module needs a context in the application in order to write their @moduledoc</code> so the feature request is to make a dependency module graph and include the dependency @moduledoc</code> or the code of dependency module in the prompt. (TO BE DONE). </li> Another interesting feature is to clean all the docs, so this way you can regenerate them from scratch, so we have created a third task mix lazy_doc.clean</code> which deletes all the @doc</code> annotations, but it keeps @doc false</code> annotations. This is useful when your product has evolved, but your docs have not evolved with your product. </li> Make a default prompt to generate docs compliant with ex_docs</code>, this way when we generate the docs webpage, you can experience the documentation as it was a usual docs. </li> </ul> Conclusion</h2> Making new projects make you learn a lot of stuff. Usually to share your projects, you need to write how to use them properly, that is the reason developers write docs. I do not know a bunch of people who likes writing docs or they have time to write them. Writing docs maybe does not add too much business value to the product you are building but it will add a huge value to developers building the product or the developers using your product and they are gonna be less prone to making mistakes. Writing LazyDoc, it allows me to learn about deep aspects of the Elixir core, Elixir AST and learn what weird stuff you can do. For example defmodule Hello do defmodule Hello do end end</code></pre> This is valid Elixir code, but it is not allowed in LazyDoc. Writing LazyDoc, I realize that good software sometimes needs to be opinionated, take a strong opinion and follow it until the end, for example external_doc</code> feature is really opinionated because some people will like to have the docs in the code and some other will just like to see the code and not being distracted but anything else. Another strong opinion that this library takes is what does with the functions of the same name, basically we understand that they are the same function to document even that for Elixir it is not. At least every teammates I asked told me that 2 functions named the same are always related, so it makes sense to document them together. Same nested named modules does not make sense in almost any situation, so LazyDoc does not allow it because they have the same AST structure. Thank you very much for your reading effort. You can support lazy_doc repo</a> and if you are curious how LazyDoc docs look, LazyDoc is documented by LazyDoc itself you can check it in lazy_doc hexdocs</a>

Fang, async background processing for Rust.

pxp9 — Sat, 06 Aug 2022 00:00:00 +0000

Even though the first stable version of Rust was released in 2015, there are still some holes in its ecosystem for solving common tasks. One of which is background processing.

In software engineering background processing is a common approach for solving several problems:

Conclusion</h2>
The project is available on GitHub</a></p>
The async feature and this post is written in collaboration between Ayrat Badykov</a> (github</a>) and Pepe Márquez Romero</a> (github</a>)</p>

Software Engineer Oasis

A gentle introduction to Elixir IoT

Why Elixir for IoT ?</h2>

Fibonacci calculation at peak performance

Lazy Doc, Your documentation tool has arrived.

Fang, async background processing for Rust.

Software Engineer Oasis

A gentle introduction to Elixir IoT

Why Elixir for IoT ?</h2>

Reduce the amount of languages in your stack.</h3> Consider you want to do a Full system from the smallest MCU chip to Cloud.</p> How would you do it without the Elixir way?</p>

Feature rich in embedded systems which you cannot imagine or not even come close.</h3> Fault tolerant. Your application can recover from a unexpected failures. This feature is builtin in any Erlang VM application.</p> </li>

What kind of stuff I built with this tech ?</h2> The reason I am writing this article is because I have built an LLM agent which controls, the hardware via prompts.</p>

Fibonacci calculation at peak performance

Math is not real life.</h3> Based on the math we have seen before we can think this algorithm:</p>

Theory pretty cool, but show me the numbers.</h3> Ok ok, now is the moment you were waiting for! The benchmarks and see if the theory is confirmed.</p>

Lazy Doc, Your documentation tool has arrived.

Fang, async background processing for Rust.

Threaded Fang</h2> Fang is a background processing library for rust. The first version of Fang was released exactly one year ago. Its key features were:</p> Each worker is started in a separate thread</li>

Pitfalls</h2> The async processing is suitable for lightweight tasks. But for heavier tasks it's advised to use one of the following approaches:</p>

Conclusion</h2> The project is available on GitHub</a></p> The async feature and this post is written in collaboration between Ayrat Badykov</a> (github</a>) and Pepe Márquez Romero</a> (github</a>)</p>

Conclusion</h2>
The project is available on GitHub</a></p>
The async feature and this post is written in collaboration between Ayrat Badykov</a> (github</a>) and Pepe Márquez Romero</a> (github</a>)</p>