The C++ Alliance

Krystian’s Q2 2024 Update

2024-07-15T00:00:00+00:00

MrDocs

This quarter, my primary focus was on MrDocs. To that end, I’m happy to say that the P0 milestone for MrDocs has been reached! Although the vast majority of the work I did was bug hunting/fixing, I also implemented a novel feature which detects and simplifies types that use the SFINAE idiom.

Prior to C++20, the primary mechanism for constraining templates was through use of the SFINAE idiom, e.g.:

template<typename T>
std::enable_if_t<std::is_pointer_v<T>, T> f(T); // only viable when T is a pointer type

These kinds of constrained declarations can be rather “noisy” due to the constraints being written into the type of the declaration. One solution is to use macros to show the simplified type when generating documentation:

template<typename T>
#ifndef DOCS
std::enable_if_t<std::is_pointer_v<T>, T>
#else
T
#endif
f(T);

but this requires manual intervention on the part of the library author. Since MrDocs generates documentation using the clang AST, we can detect whether a template implements the SFINAE idiom and show the “replacement type” instead:

template<typename T>
std::enable_if_t<std::is_pointer_v<T>, T> // rendered as 'T' in the documentation since std::enable_if_t is a SFINAE template
f(T);

In general, MrDocs determines whether a type naming a member M of a class template C uses the SFINAE idiom by identifying every declaration of M in the primary template and every partial/explicit specialization of C. If M is not found in at least one of the class definitions, and if every declaration of M declares it as an alias for the same type as every other declaration, then C is a template implementing the SFINAE.

Clang

In addition to working on MrDocs, I merged a number of clang patches this quarter:

Braden’s Q2 2024 Update

2024-07-14T00:00:00+00:00

Speaking at C++Now

At C++Now 2024, I gave my first full-length conference talk, titled “Unit testing an expression template library in C++20”. The slides are up at this link. As of the time of writing, the talk has not yet been uploaded to YouTube.

In this talk, I explored the current state of compile-time unit testing in C++ in the well-known unit testing libraries. I also discussed my own methods for unit testing at compile-time. This talk especially focuses on giving helpful diagnostics when encountering an error at compile-time. I used my tok3n library (a personal project) as a backdrop for the testing, which has been the driving force behind exploring this area in the first place.

I had an enjoyable time preparing and giving this talk, and I certainly learned a lot. I appreciate the support from the others at the C++ Alliance, for helping me to grow as a speaker and to have this amazing opportunity to speak at C++Now 2024.

Boost.Unordered Visual Studio Natvis support

In Q2 2024, I created the “boost_unordered.natvis” file, to give user-friendly visualizations for all Boost.Unordered containers in the Visual Studio debugger. In doing this, I learned quite a lot about Natvis files, which I wrote about in this article, titled “Natvis for boost::unordered_map, and how to use elements".

Unfortunately, I wasn’t initially able to support all Boost.Unordered containers. I thought I had to exclude the containers with allocators that use fancy pointers (like boost::interprocess::offset_ptr), but I consulted with Joaquín to eventually find a solution. It didn’t technically make it for Q2, but it’s close enough. I wrote a 2nd article explaining this whole process.

I am excited to continue improving the user experience for Boost.Unordered. Next up, I’ll be tackling GDB pretty-printers for the containers.

Alan’s Q2 Update 2024

2024-07-13T00:00:00+00:00

Summary

MrDocs
Boost Libraries
Boost Release Tools
Boost Website
C++ Github Actions

MrDocs

MrDocs is a tool for generating reference documentation from C++ code and javadoc comments. I have been overseeing and reviewing all contributions to the project.

Krystian has focused on metadata extraction and issues related to Clang.
Fernando has been tackling usability, critical bugs, and essential features.
I’m enhancing the CI setup and working on Antora extensions, which will be incorporated into website-v2-docs.

Currently, metadata extraction primarily involves identifying SFINAE techniques in the code, enabling MrDocs to natively comprehend C++ constructs.

In the last quarter, I authored several documents comparing MrDocs to its alternatives:

Gap Analysis: Compares MrDocs to alternatives, identifying and prioritizing missing features.
Minimum Viable Product (MVP): Outlines the features necessary for MrDocs to be competitive.
Product Pitch: Explains MrDocs’ value proposition.

The MVP aims to include features common to all alternative tools, preventing initial adoption blockers and ensuring a positive first impression. Additionally, the MVP includes unique features of MrDocs, such as native comprehension of C++ constructs. We identified numerous selling points for MrDocs, including maintainability, multiple output formats, and its understanding of C++ constructs.

We created a GitHub project to track MrDocs’ progress, with issues categorized by priority:

P0 - Boost Features: Critical bugs and usability issues, using Boost.URL as a proof of concept.
P1 - Main MVP Features: Essential MVP features requested by users and Boost projects.
P2 - Technical Debt: Technical debt from P0 and P1.
P3 - All MVP Features: All other MVP features.
P4 - Non-MVP Features: Features beyond the MVP outlined in the gap analysis.

P0 encompasses all issues necessary for MrDocs to be utilized by an Antora extension in the Boost Release Tools. Using Boost.URL as a proof of concept, our goal in P0 is to replace the Doxygen + Docca workflow with MrDocs. We completed P0 recently, with partial results available at https://843.urlantora.prtest2.cppalliance.org/site/url/index.html

As we transition to P1, I’m adapting the Boost.URL project with Antora extensions to include reference documentation generated by MrDocs in the website-v2-docs Antora project. P1 includes all features identified in our Gap Analysis for MrDocs to be competitive with alternatives. We are also improving usability and documentation based on feedback to ensure users can effectively utilize the tool. Completing P0 and P1 will accumulate enough technical debt to start addressing P2.

In Q2/2024, we implemented several features and fixes, including:

Detect SFINAE idiom
Extract using directives, declarations, namespace aliases
Extract pack expansions in base-specifiers
Extract variadic functions
Extract explicit specifiers
Detect implementation-defined and see-below namespaces
Support @pre, @post, and @n commands
Annotate templates with See Also sections, constructors, and destructors
Ensure HTML templates match AsciiDoc templates
Implement select handlebars helper
Link names in overload sets
Include full nested name specifiers in titles
Ensure safe names for section IDs
Provide warnings for duplicate commands
Propagate dependency modes
Extract dependencies for templates
Check for atomic shared pointer support
Update XML schema
Escape names of overload set members
Restore HTML demos
Normalize source-root
Ensure global namespace is always present
Remove symbol type from titles
Update LLVM version to 7a28a5b3 and adapt the project
Treat references to purposefully excluded symbols as non-errors
Emit error messages for invalid javadoc commands
Manage info and config tables as .inc files
Print diffs in golden tests
Create CI demo variants with command line variants
Build statically for MrDocs to be used in CI by other projects
Use default containers for each compiler
Update CI to use official releases for dependencies, enhancing stability
Compare generated documentation in PRs with a demo comparison artifact
Include ref_name in demo artifact names for proper PR comparison
Upload documentation to the website
Ensure template consistency with checks
Split test targets
Set default build type to RelWithDebInfo
Clean headers and footers in docs
Create a demo page in the documentation using a custom Antora plugin
Add an install page with direct links to GitHub releases
Provide more extensive usage examples
Complete the configuration file reference
Include design notes on rationale and philosophy
Create a Contributor Guide
Refer to official documentation in README.adoc
Add a new banner referring to Mr. Docs

Integrations

As Boost.URL continues to integrate MrDocs, it has inspired the necessary features in MrDocs and is temporarily generating documentation with both Doxygen+Docca and Antora+MrDocs. The same workflow is implemented in Boost.Buffers, inspiring new feature requests such as identifying Niebloids and SFINAE constraints in documentation.

The features implemented in Boost.URL are described in the Boost Libraries section.

General work

Overall, my responsibilities in MrDocs include:

Setting up and maintaining CI for the project;
MrDocs and LLVM release binaries;
Build scripts;
Setting up and integrating dependencies;
Setting up and deploying the Antora toolchains and documentation to the project website;
Working on supporting libraries;
Supervising and reviewing the work done by other contributors (Krystian and Fernando); and
Fixing bugs.

Boost Libraries

The Boost library I’ve dedicated the most time to is Boost.URL. The library is in maintenance mode, but there is a constant demand for bug fixes and documentation improvements. Recent commits focus on the Antora+MrDocs workflow:

MrDocs collector plugin identifies versioned compiler executables on the host machine
Documentation includes a manual reference table
Refactored source code to distinguish details, implementation-defined, and see-below symbols
Created a new MrDocs target that includes all headers, reducing MrDocs run time from ~8 minutes to ~3 seconds and ensuring all headers are included in the documentation

Other significant updates in CI and documentation include:

CI: Added support for Node.js 20 actions due to GitHub updating the base Node.js version to 20. The lack of static linking in Node.js caused errors related to glibc versions.
CI: Updated the Antora workflow to use Clang 18, addressing issues with standard library synchrony between the host machine and LLVM/Clang versions.
Docs: string_token exposition
CI improvements were coordinated with the C++ Github Actions project, which implemented new features for these scenarios.

Boost Release Tools

I have integrated the toolchains I developed into the Boost Release Tools, adding support for features desired for the new Boost website, including the Antora+MrDocs workflow.

Since the last report, we adapted the documentation workflow to relocate the Antora documentation in the release. Additionally, Boost.URL documentation in the Antora format will be included in the release once we enable it in website-v2-docs. We are delaying this to avoid disruptions during the current release cycle. Meanwhile, we continue using the Doxygen+Docca workflow and are developing the Antora UI for the next release.

Boost Website

Among support projects for the new Boost website, I have been particularly involved with website-v2-docs, which includes the Boost website documentation as an Antora project. Its components cover the “User Guide,” “Contributor Guide,” and “Formal Review” sections.

Since the project’s inception, I have been overseeing and reviewing contributions from other team members.

Overall, my responsibilities include:

Reviewing and merging pull requests
Setting up and maintaining CI
Coordinating with the website project on content uploaded to AWS buckets
Developing reusable build scripts for release tools and previews
Writing technical sections of the documentation
Developing custom Boost/Antora extensions, such as the Boost Macro extension
Maintaining the Antora toolchain and templates
Adjusting Boost libraries to match the expected formats

For the next quarter, we plan to update the Antora UI bundle to resemble the current Boost website style.

C++ Github Actions

C++ Github Actions is a project I have maintained since 2023. It is a collection of composable, independent, and reusable GitHub Actions for any C++ project needing testing on various compilers and environments.

MrDocs, Boost.URL, Boost.HTTP, and Boost.Buffers currently use these actions in their CI. These projects provide valuable feedback to improve the actions.

The project includes actions to:

Generate a Github Actions Matrix for C++ projects;
Setup C++ compilers;
Install and setup packages;
Clone Boost modules;
Run complete CMake and b2 workflows;
Generate changelogs from conventional commits;
Generate summaries; and
Generate time-trace reports and flame graphs

The actions are designed to be modular and interoperable. Each action has a specific role, such as configuring an environment or building a C++ project. They can be composed to create customized CI/CD workflows.

One notable problem with GitHub actions in Q1/2024 is that GitHub decided to update the base version of Node.js to 20. Because Node.Js is not statically linked, this change broke many actions that depended on it. All projects that depended on older containers were getting an error related to the appropriate glic version not being found. Not updating to Node.js 20 would also give users deprecation warnings.

We ended up implementing a solution where older containers suggested by the cpp-matrix action have a volumes key to create a directory where node can be installed in such a way that GitHub actions will use this node executable. If using an old container, the user is responsible for installing Node.js in that directory. Examples are provided in the repository.

Thus, a number of new features and fixes were added to the C++ Actions project in Q2/2024.

Feature: support clang >=18 (which is also critical for MrDocs workflows)
Feature: cpp-matrix force factor flags
Feature: cpp-matrix is-container auxiliary key
Feature: cpp-matrix older containers suggest volumes for node 20
Fix: cpp-matrix windows runner has MSVC 14.40.33810
Fix: cpp-matrix default macOS is macos-14
Fix: setup-gcc GCC 14 comes from Ubuntu 24.04
Fix: setup-gcc ensure software-properties-common
Fix: setup-clang verify LLVM repo release files to filter repositories that do not exist
Refactor: package-install set DEBIAN_FRONTEND to noninteractive
Refactor: cpp-matrix extract support sanitizer as boolean
Refactor: all actions use node20
Refactor: cpp-matrix pkg-config included in container default installs
CI: only create releases for tags
CI: external actions updated
CI: matrix.cxx not required
CI: b2-workflow variant2 does not test on windows
CI: older containers patch node
CI: update actions/checkout to v4
Test: cpp-matrix custom suggestions
Docs: cpp-matrix open ranges

Most work has been done to fix issues revealed by testing the actions in new environments.

Kenneth’s Q2 2024 Update

2024-07-12T00:00:00+00:00

Project Overview: Boost.io Website Development

During the second quarter of 2024, my primary focus has been on the development and enhancement of the new boost.io website. This project has been a significant undertaking, aimed at improving the online presence of the Boost C++ Libraries and providing a more user-friendly experience for our community.

Key Accomplishments:

Onboarding to the C++ Alliance and Boost.io Repository

Being a newer member of the team, I’ve been getting onboarded onto the website-v2 repo and helping other new members of the team get onboarded as well.

Thankfully, the code quality in this repository is pretty high, so there’s little concern in this department.

Website Usability

I spent a lot of time this quarter improving the usability of the website in subtle ways. Cleaning up the navigation, library titles, adding compatibility redirects to support URLs that were live on the [boost.org] website, etc.

Developer Experience

While there remains a lot of work in this area, I’ve done a few things like integrating uv into the build process to help speed up CI builds. In the future, I plan to use its pip compile functionality, too.

Documentation Experience Improvements

I spent a reasonable amount of time this quarter focusing on improving the navigation and usability of the boost.io website’s support of arbitrary documentation.

Some clever improvements have been put in place to ensure that the user experience remains ideal as we continue to improve the solution, overall.

Hosting documentation is not without its complexities.

Looking Forward:

Working on this project has deepened my understanding of Django and web development best practices. The collaborative nature of the C++ Alliance, especially in its remote setting, has been instrumental in overcoming challenges and driving innovation in our approach to presenting Boost libraries to the community.

I look forward to continuing to work on these projects and enjoy helping to make the world a better place!

Dmitry’s Q2 2024 Update

2024-07-12T00:00:00+00:00

In the second quarter of 2024 implementing direct parsing for Boost.JSON has finally been completed. Direct serialisation will support all the same types as direct parsing, including recently-added std::path. After this addition, Boost.JSON’s set of conversion features is almost full. The only missing part is the ability to use a sort of temporary proxy type for conversion. E.g. converting from user type to std::string, then converting the string to JSON string. This is not strictly needed for value_to/from set of functions, as you can always fully customise their behaviour by extending them. But it can be very useful for direct parsing, as it allows using an ancillary type that matches the structure of the JSON, and can convert to/from the intended user type.

A little more than a year ago it was argued in a Boost.JSON issue that object::if_contains should return optional<value&> rather than value*. I have experimented with that, but also went on to research whether there are any projects on GitHub that would be broken due to such change. Turns out there’s a lot of such projects. Which is why I decided against that change. But on the other hand, there are some benefits to having non-throwing accessors that communicate their inability to produce a value in their return value. And it’s better to use a single kind of type for this. There is in fact an ideal candidate for such return type: boost::system::result. And in fact Boost.URL is using it in its return types extensively, eschewing the classic dual API approach (where one overload throws, and another overload uses an error_code& out parameter). If I could turn back time, I would have replaced the out-parameter overloads with ones returning boost::system::result. But as that would be way too significant API change now, I instead added those accessors, giving them try_ prefix (modelled after the already existing try_value_to). As a tangentially-related change, I added a defaulted source_location parameter to throwing accessor functions. This is a small quality of life improvement, that (if supported by the compiler) stores the location of the call site in the exception. The result is that the exception stores both the line where the user calls a Boost.JSON function, and (inside the error_code) the line where the error state was originally reached. This information should greatly simplify investigating issues that occur when using the library.

A significant amount of time these past few months was occupied by a somewhat new project: Python-based reimplementation of Docca. Docca is a set of XSLT stylesheets that converts Doxygen XML output into an API reference written in Quickbook markup language. Unfortunately, XSLT is both rather obscure, and rather cryptic, which results in difficulty fixing its bugs. As my frustrations piled up, I decided to write a new tool that would essentially do the same job, but in a more popular language, and designed for higher genericity. I chose Python for implementation largely due to its Jinja text template engine (another reason is Python’s availability on virtually all platforms).

The core of the design is operating in two steps: 1) collecting data from Doxygen XML output, organising it in a usable way, and sometimes even fixing Doxygen’s warts and 2) generating output using Jinja. This kind of model/view separation isn’t just philosophically more pleasant, but also makes the tool more generally useful. As previously mentioned Docca produces a specific style of API reference written in Quickbook. But this new implementation can use a different Jinja template to produce a different style of API reference: e.g. using another approach to sectioning, or having all overloads on one page a-la Cppreference. It can also produce output in an entirely different format, and I intend to take advantage of that in order to switch Boost.JSON’s documentation to Asciidoc. I am also experimenting with running the tool an additional time to generate “macros” that expand into links to documented entities and could be used in the narrative documentation that precedes the reference.

On the opposite side of this two phase arrangement is the ability to add alternative data collection algorithms. While currently Doxygen reigns supreme in the field of collecting documentation comments from C++ sources, there is a very ambitious project from the C++ Alliance called Mr. Docs. As soon as it reaches enough maturity, a new data collector could be added to Docca.

And finally, I wanted to tell about some changes I’ve contributed to Boost’s build infrastructure, but I need to start from a far. The build system officially supported by Boost is called b2 (formally known as Boost.Build). Unlike most modern build systems it doesn’t use the 2 step configure-build approach championed by Autotools, and instead does the whole build by itself. As a result, it’s a fully multi-config build system, that is a single invocation of the build system can request targets built with different values for particular property, resulting in building of multiple variations of those targets. For example, you can request a target to be built by different compilers, targeting different OSes and architectures, using different C++ standards and dialects, and so on. b2 would calculate the Cartesian product of all requested options, and create the resulting binaries for each of the variations. For historical reasons, there was no good support for this multi-config paradigm with installation. install targets explicitly specified their intended installation location, which resulted in build errors if a multi-config installation was attempted. The issue wasn’t fundamental to the build system, and Boost had employed a particular mitigation to that issue. But I wasn’t satisfied with the status quo, so a few years ago I added explicit support for named installation directories to the install rule, with those directories’ configuration being a part of the build variation. After the change a project could do something like

exe app : app.cpp ; # executable target app
install install : app : <location>(bindir) ; # install target install

And then with b2 install-prefix=/usr/local install app into /usr/local/bin, and with b2 install-prefix=/opt/myapp install app into /opt/myapp/bin. You could then go with a conditional property configuration, where e.g. target OS a implies location 1, and OS b requires location 2, and so on.

The feature did require changing build scripts, and so at the time Boost remained with the set up it already had. But this spring, inspired by other needs I finally got around to changing Boost.Install (an ancillary project used by Boost’s build scripts) to use this functionality. One thing led to another, and now not only users can configure per-config installation directories, but finally Boost has staging directory support via staging-prefix. E.g. b2 --prefix=/opt/Boost/ staging-prefix=/var/tmp/1 installs into /var/tmp/1, but files are created with the intention to be later moved to /opt/Boost/.

Christian’s Q2 2024 Update

2024-07-10T00:00:00+00:00

Boost.Http.Proto

This last quarter I worked primarily in the http-proto library, this time extending serialization to include chunking and zlib compression routines: deflate and gzip.

This is the first time I’ve ever used zlib so it was definitely a learning experience but it’s given me valuable insights into how other libraries of this nature work.

More importantly was reifying the application of such dynamic code with the existing code which supports many possible permutations.

Serialization in http-proto enables users to consume output in myriad different ways and the output itself can be framed and transformed as well. This forms a product space, in all actuality, so it was quite a feat to unify the core logic. The secret was operating in terms of distinct input and output buffer spaces, which could sometimes alias.

Boost.Compat

I also dedicated some time into working on Boost’s Compat module, which is home to several different kinds of polyfills of C++ constructs introduced in later C++ standards.

This time I worked on function_ref and as far as I’m aware, I’m the first implementor of such a facility. The class seems very simple on the surface. Just a simple non-owning type-erased view of a Callable object but the devil is always in the details.

const and noexcept each change the actual type of the function signature used so again, I had to test another proudct space. The testing burden for many of these components is quite high and while there are some, it’ll be interesting to see if anyone actually uses this facility and how well it fares.

Interestingly, function_ref actually has a form of currying with regards to member access and objects. This doesn’t seem well-known and actually came as a surprise to me.

There’s a lot more to look forward to later in the year and I’ll be excited to write the next update.

Christian

Mohammad’s Q2 2024 Update

2024-07-10T00:00:00+00:00

The following is an overview of some projects I have been working on in the last few months:

Boost.Beast

Besides addressing user issues and typical bug fixes, here are a couple of contributions to the Boost.Beast project that I find interesting:

Deprecating `beast::ssl_stream` and `beast::flat_stream`

The beast::flat_stream was originally designed to consolidate buffer sequences into a single intermediate buffer, reducing the number of I/O operations required when writing these sequences to a stream that lacks scatter/gather support. However, after Asio improved the performance of write operations in ssl::stream by linearizing gather-write buffer sequences, the necessity for beast::ssl_stream and beast::flat_stream has decreased. Their continued inclusion has caused confusion among new users and added unnecessary complexity to the documentation and code snippets. Consequently, these streams are now deprecated, and the use of asio::ssl::stream is recommended in the documentation, examples, and code snippets.

Note: Existing code will remain functional, as beast::ssl_stream has been redefined as a type that publicly inherits from asio::ssl::stream.

Simplifying C++20 Coroutine Examples

The latest Asio release introduces several changes that significantly reduce code complexity and noise. Previously, we had to redefine IO object types with an executor that has a default completion token type, like this:

using tcp_socket = asio::as_tuple_t<asio::deferred_t>::as_default_on_t<asio::ip::tcp::socket>;

With the new Asio release, asio::deferred is now the default completion token, eliminating the need to redefine IO objects with custom executors. Now, these work out of the box:

auto n = co_await http::async_read(stream, buffer, res);
// Or with a partial completion token:
auto [ec, n] = co_await http::async_read(stream, buffer, res, asio::as_tuple);

Using these features and the new asio::cancel_after functionality, all the C++20 coroutine examples in Beast have been refactored to be more concise and clear. Additionally, a task_group has been added to the advanced_server_flex_awaitable example to demonstrate a graceful shutdown process by propagating a cancellation signal to all session subtasks.

Adding New Fuzzing Targets

In this release, we addressed two bug reports that were caught by fuzzing the code: #2879 and #2861. Thanks to tyler92, we have added several fuzzing targets to the project. These targets now fuzz the code with each pull request and at scheduled times throughout the day.

Boost.Http.IO

As the Http.Proto project continues to mature, we have begun enhancing the Http.Io examples to ensure our sans-io API aligns closely with the evolving requirements of the I/O layer. To start, I have introduced a basic C++20 coroutine client example, which will later be expanded into a tool similar to curl.

Boost-Gecko

Boost-Gecko now features a new index for the learn section of the updated Boost website and a new crawler for website-v2-docs. This update allows users to search within the documentation in the learn section using the website’s search dialog. The crawling process and index record uploads are automated through a GitHub workflow, requiring minimal maintenance for each Boost release.

Sam’s Q2 2024 Update

2024-07-10T00:00:00+00:00

Here’s an overview of some projects I have been working on the last few months.

Boost Downloads

Discuss the details of uploading windows builds to the new CDN with Tom Kent. When publishing the boost releases, they are first uploaded to an S3 bucket s3://boost-archives/. Then the CDN origin servers download the files locally.

Boost website boostorg/website-v2

A Fastly CDN is configured in front of the boost.io website to increase performance and reduce download times. Fastly is hosted version of Varnish software and uses the VCL programming language. I have been participating on their community forums to find solutions and techniques to adjusting headers, removing cookies, modify cache settings. The VCL framework allows manipulating the http traffic at various phases of the packet’s journey during recv, pass, fetch, hit, miss. The language is distinct, but similar to javascript or python type languages. Multiple iterations of adjusting the VCL. This is an ongoing project. Removed csrf tokens from certain website pages to facilitate caching.

Configuring oauth social auth for the website domains, continued from previous months. Remove fallback default url from boost.io, to reduce traffic on S3, accelerate response times.

Upload boost release notes to S3 so boost.io is able to parse antora markup files when switching from Quickbook to Antora.

Launched a server regression.boost.io, an AWS EC2 instance to host uploaded boost regression test reports on the new website.

Created a static html mirror of boost.org: boost.org.cpp.al.

boostorg/release-tools

Developed an autocancel feature in boostorg/boost CircleCI to prevent multiple boost release builds from conflicting.

Multiple updates to release-tools to support the new Fastly workflow.

Switch all boost builds to use python virtual environments (python pip packages).

Mailman project

Sent an email to the boost list, explaining the mailman3 project and requesting testers. Based on their feedback, researching and solving some issues in the mailman deployment at lists.preview.boost.org. Discovered an apparent bug in the mailman configuration where the home page was not immediately updating after imports or deletions. It turns out another component ‘qcluster’ must be running. Adjust installation scripts accordingly. Sent a (merged) pull request to upstream hyperkitty, fixing a formatting issue.

wowbagger

A wowbagger disk failed on two different occasions, causing an outage of the boost.org website. Worked with David Sankel to restore the disk, recover from errors. Updated the legacy boost.org website to use Fastly archive downloads. This involved modifying numerous pages on the site and including new functions in boost-tasks.

Migrated the boostorg/boost commitbot from wowbagger to Github Actions.

Jenkins

New Antora doc previews for buffers, http-io, http-proto. An outstanding project with Revsys is to revamp the lcov/gcovr code coverage displays. With that in mind I have cleaned up and rewritten the gcovr scripts that Jenkins uses so those are streamlined and easier to work with in the future.

LLVM

Meetings with Google about LLVM CI. The Alliance suggested funding an improvement in their CI infrastructure. This interaction encouraged Google to become more involved, as they had previously committed to support the clang llvm project financially. Either way, hopefully the end result will be faster LLVM CI builds.

GHA

New 24.04 VM images.

cpp.al

Enhanced CircleCI jobs to remove concurrency and force sequential processing/rendering of blog posts on the cppalliance.org website.

Drone

Assist in debugging cppalliance/decimal, charconv, drone jobs. Publish new 24.04 drone containers. Upgrade auto-scaled VMs also. Configured swap space (disk-based memory) on auto-scaled drone agents. Refactor a few steps in the script that builds the drone server image.

Ruben’s Q2 2024 Update

2024-07-09T00:00:00+00:00

On C++20 modules and Boost

This quarter started with exciting discussions about the possibility to introduce C++20 modules in Boost. I’ve dedicated a lot of time to study and reduce Boost.MySQL build times, so I promptly volunteered to conduct some investigation on the benefits and costs of modules.

I’ve written two articles (available here and here) about this topic. They can be roughly summed up as:

Module clean builds aren’t as fast as I’d expect, but partial re-builds are much faster.
Modules are still poorly supported by compilers and tooling, although work is being performed on them.
Making a library consumable as a module requires non-trivial development, testing and maintenance effort.
At the point of writing, mixing standard includes and imports causes problems (although the standard mandates the opposite). As a consequence, if we’re to support consuming a certain Boost library as a module, all its dependencies must support it, too.
At the point of writing, most of the Boost community considers the effort is too big, and prefers to wait until modules are more stable.
Distributing modules so they can be consumed by CMake is highly non-trivial and one of the biggest blockers.

I’m happy for this discussion to have taken place. We’ve all learnt a lot, and we now know enough to make informed decisions.

using std::cpp 2024

I’ve had the honor of getting a talk on Boost.Asio’s universal async model accepted for using std::cpp 2024. You can find the slides and code samples here.

The best part of it has been getting in touch with the community. I’ve met a lot of real Asio users and been able to discuss their pain-points in person. I’m happy to see most people working with C++17 and C++20 rather than old standards. Many people manifested interest in C++20 coroutines, but are not willing to roll their own awaitable types - they just wanted to call co_await and go. I was surprised to learn that they didn’t know that asio::deferred already could do this for them, so I think I got the talk topic right.

I think we need to lead by example, and I’d like to re-write servertech-chat using C++20 coroutines - it’s a great example for newcomers to learn these.

I’ve also been answering many of the questions coming up in #boost-asio in Slack.

Client-side SQL formatting enhancements

Boost.MySQL client-side SQL allows composing queries client-side without incurring in injection vulnerabilities. It works great for simple queries, but it was too verbose for cases involving ranges. Consider batch lookup:

// Retrieve all users matching the IDs provided in ids
asio::awaitable<std::vector<user>> lookup_users(any_connection& conn, std::span<const std::int64_t> ids)
{
    // Compose the query
    mysql::format_context ctx(conn.format_opts().value());
    ctx.append_raw("SELECT * FROM user WHERE id IN (");
    bool is_first = true;
    for (auto id: ids)
    {
        // Comma separator
        if (!is_first) ctx.append_raw(", ");
        is_first = false;

        // Actual id
        mysql::format_sql_to(ctx, "{}", id);
    }
    ctx.append_raw(")");
    std::string query = std::move(ctx).get().value();

    // Run it
    mysql::static_results<user> res;
    co_await conn.async_execute(query, res);
    co_return {res.rows().begin(), res.rows().end()};
}

That’s verbose and easy to get wrong. And the price of an error here can be a vulnerability. To solve this, we’ve added built-in support for ranges:

asio::awaitable<std::vector<user>> lookup_users(any_connection& conn, std::span<const std::int64_t> ids)
{
    // Compose the query. May generate "SELECT * FROM user WHERE id IN (10, 21, 202)"
    auto query = mysql::format_sql(conn.format_opts().value(), "SELECT * FROM user WHERE id IN ({})", ids);

    // Run it
    mysql::static_results<user> res;
    co_await conn.async_execute(query, res);
    co_return {res.rows().begin(), res.rows().end()};
}

Much better, isn’t it? And if you need additional functionality, mysql::sequence allows to pass custom glue strings and per-element formatting functions. Most cases, including batch inserts, can be expressed in terms of a single format string.

Our next step here is implementing an easy-to-use execution request that colaesces composing the query and executing it in a single step. This came up during the review, and it’s finally going to be a reality.

Pipeline mode

MySQL client/server protocol is strictly half duplex. The client sends a request, and the server responds. Performing some measuring, some use cases involving lightweight requests are dominated by round-trip time. In these cases, coalescing individual requests into a single message helps performance.

Use cases fitting this description include connection setup code and preparing/closing statements in batch.

The connection_pool class has been using this feature internally for a release, and we now expose it for the general public:

// Sets up a connection for re-use. connection_pool cleans up connections in a similar way
asio::awaitable<void> setup_connection(any_connection& conn)
{
    // Build a pipeline describing what to do.
    mysql::pipeline_request req;
    req.add_reset_connection() // wipe session state
        .add_set_character_set(mysql::utf8mb4_charset) // SET NAMES utf8mb4
        .add_execute("SET time_zone = '+00:00'"); // Use UTC as time zone
    std::vector<mysql::stage_response> res;

    // Execute the pipeline
    co_await conn.async_run_pipeline(req, res, asio::deferred);
}

We can write this because we fully control the library’s serialization and networking, rather than wrapping other MySQL clients.

Other Boost.MySQL work

I’ve also worked on lesser (but necessary) tasks on Boost.MySQL, including enabling buffer size limits for any_connection, adding support for C++20 time types to our date and datetime types, as well as some bug fixing and refactoring.

Fernando’s Q2 2024 Update

2024-07-08T00:00:00+00:00

As we move through 2024, I have continued to focus on enhancing the capabilities of MrDocs, addressing key features and fixes to push towards the Minimum Viable Product (MVP). Additionally, I have been exploring profiling tools and developing benchmarking sets to optimize the performance of concurrent data structures in Boost.

MrDocs Development: Aligned with MVP Strategy

This quarter, my efforts on MrDocs have been centered around resolving crucial bugs and aligning features essential for the MVP. The following are some notable contributions:

Bug Fixes and Template Alignment:
- PR #572: Fixed bug reporters referring to MrDocs repository. This was essential for improving user engagement and error reporting. View PR
- PR #604: Addressed alignment and CI issues in the HTML template generator, which was a significant update involving modifications across 60 files. View PR
- PR #632: Fixed a crash when processing empty Translation Units (TUs), ensuring stability across all scenarios. View PR
- PR #633: Synchronized AsciiDoc and HTML templates to maintain consistency across documentation outputs. View PR
- PR #640: Applied East-const style to documentation generation to enhance code readability and maintenance. View PR

These enhancements not only address immediate functional and stability needs but also align with the broader goals of making MrDocs competitive in the documentation generation landscape, in accordance with the strategic focus on the MVP.

Boost Concurrent Flat Map

As part of my ongoing contributions to the Boost Libraries, my current focus has been on enhancing the understanding and performance capabilities of the Boost Concurrent Flat Map. This process involves intensive measurement and the development of a comprehensive benchmarking suite:

Performance Profiling and Benchmarking:
- Utilizing AMD μProf: I have begun employing AMD μProf on an AMD Threadripper with 64 cores for in-depth performance profiling. This tool is instrumental in identifying critical performance bottlenecks that can inform future optimizations.
- Creating Real-World Benchmarks: To ensure the relevance and effectiveness of our performance enhancements, I am developing a benchmark suite using real data from the Bitcoin blockchain. This data is particularly challenging and representative of real-world high-load scenarios, making it an excellent basis for rigorous testing.
- Benchmark Goals: The primary aim of these benchmarks is to measure the performance of Boost Concurrent Flat Map in the context of implementing critical Bitcoin infrastructure components such as the Mempool and the UTXO Set Module. These components are crucial for transaction validation and processing, thus requiring efficient and robust data handling capabilities.

This focused approach on profiling and setting up a real-world benchmarking environment is crucial for laying the groundwork for future enhancements. By understanding the current performance levels under realistic conditions, we can better tailor the Boost Concurrent Flat Map to meet the demanding needs of modern data processing tasks.

Reflections on Remote Collaboration and Open Source Contribution

Working asynchronously on these projects across different time zones continues to offer a unique set of challenges and opportunities. The collaborative nature of open-source work with the C++ Alliance enriches my professional experience and contributes significantly to my personal growth.

Looking Ahead

As we move into the next quarter, my focus will remain on advancing MrDocs towards its MVP and continuing to explore profiling tools and benchmarking strategies to enhance the efficiency of Boost libraries. The journey of innovation and improvement is ongoing, and I am excited about the potential developments in the coming months.

The C++ Alliance

Krystian’s Q2 2024 Update

MrDocs

Clang

Braden’s Q2 2024 Update

Speaking at C++Now

Boost.Unordered Visual Studio Natvis support

Alan’s Q2 Update 2024

Summary

MrDocs

Integrations

General work

Boost Libraries

Boost Release Tools

Boost Website

C++ Github Actions

Kenneth’s Q2 2024 Update

Project Overview: Boost.io Website Development

Key Accomplishments:

Onboarding to the C++ Alliance and Boost.io Repository

Website Usability

Developer Experience

Documentation Experience Improvements

Looking Forward:

Dmitry’s Q2 2024 Update

Christian’s Q2 2024 Update

Boost.Http.Proto

Boost.Compat

Mohammad’s Q2 2024 Update

Boost.Beast

Deprecating beast::ssl_stream and beast::flat_stream

Simplifying C++20 Coroutine Examples

Adding New Fuzzing Targets

Boost.Http.IO

Boost-Gecko

Sam’s Q2 2024 Update

Boost Downloads

Boost website boostorg/website-v2

boostorg/release-tools

Mailman project

wowbagger

Jenkins

LLVM

GHA

cpp.al

Drone

Ruben’s Q2 2024 Update

On C++20 modules and Boost

using std::cpp 2024

Client-side SQL formatting enhancements

Pipeline mode

Other Boost.MySQL work

Fernando’s Q2 2024 Update

MrDocs Development: Aligned with MVP Strategy

Boost Concurrent Flat Map

Reflections on Remote Collaboration and Open Source Contribution

Looking Ahead

Deprecating `beast::ssl_stream` and `beast::flat_stream`