Using Samba libraries outside Samba

SambaXP 2015

Jakub Hrozek

About me and the talk

Red Hat developer, working on SSSD full time

I'm not a Samba developer

This talk is not about Samba

What is the talk about?

This talk is about the libraries Samba provides

What is the talk about?

This talk is about...

...what it is like to maintain Samba libraries from outside Samba

...what it is like to use Samba libraries in a large project

...some interesting ways we're using the Samba libraries

...some ways we can extend the Samba libraries

It's about maintaining the Samba libraries as independent packages in Fedora and RHEL. It's about our experience using the Samba libraries in a large project that is totally independent on Samba. These two topics are first part of the presentation, maybe a bit less technical than the rest. Then, I'd I'd like to show you an interesting way we've used the Samba libraries that are not present in Samba itself, to provide access control using windows group policies. And finally, some ways we'd like to extend the LDB with a new back end and make it easier to add new back ends in the future.

SSSD

System daemon, providing identity information, authentication and access control and for remote accounts

Supports LDAP, Kerberos, Active Directory and FreeIPA

Fairly large project, about 220 kSLOC (sloccount(1))

To put the talk into context, let me very briefly introduce SSSD. SSSD is a system deamon that, with the help of Name Service Switch and PAM modules, provides access to users and groups stored remotely, in centralized databases. Currently SSSD supports LDAP for both id and auth, Kerberos for authentication. SSSD also supports FreeIPA and Active Directory a client, to support the domain member role. It's a fairly large project, totalling around 220.000 lines of code, the vast majority being written in C.

SSSD and Samba

Heavy user of Samba code

"Standing on the shoulders of Samba"

talloc
tevent
ldb
tdb
libndr, libndr-nbt
libsmbclient
cwrap: nss_wrapper, uid_wrapper, socket_wrapper

Because SSSD was started by Samba developers (Simo and Sumit), it was a heavy users of libraries originating in Samba for a long time. We use talloc, tevent, ldb and tdb in mostly predictable ways. Starting with the 1.12 version, we're also using the ndr and the libsmbclient libraries to fetch and use GPOs to provide access control. And we also started using most of the libraries that are now also available separately under the cwrap.org umbrella to provide better unit tests. Because the cwrap libraries originate at Samba and made their way back to the Samba tree, they are worth mentioning.

Samba libraries in Fedora/RHEL

In Fedora/RHEL, the separate libraries have been maintained by the SSSD maintainer, not the Samba maintainer

Historical development. Let's blame Simo!

For the most part, it's been a smooth ride, but..

Funnily, even though I'm not a Samba developer, I'm the person who maintains the Samba libraries in Fedora and RHEL. This is for historical reasons, because Simo used to be both Samba maintainer and SSSD maintainer in RHEL and even after he moved on, the Samba libraries were always maintained by whoever owned SSSD at that time. It may not sound like the most logical solution, often I get pinged by Samba developers to include a fix or build a new package to fix some of their problems that I don't understand, but at least I can give you an insight on what it is like to maintain the libraries without any Samba knowledge. For the most part, it has been easy, but there are things that can be improved.

Maintaining Samba libraries

As a maintainer, I care about changes

Is this package version safe for a stable Fedora?

Is it a bugfix-only release or are there new features?

Using Samba libraries

SSSD was started by Simo

More developers are hired to work on the SSSD

Documentation is important in getting them up to speed

Result - tevent and talloc tutorials

Documentation and tracking changes is also the only comment about how the Samba libraries are used that I have. Even though SSSD is an external consumer, we didn't have many issues in the beginning with the lack of documentation. Several developers already had experience with Samba and the others started about the same time, so we learned stuff that was crucial but not documented anywhere. But the SSSD team grows and changes and we had to teach new developers who had no prior experience with Samba. To make that easier and not repeat ourselves, developers from our team submitted or tutored documentation for talloc and tevent in a tutorial form. But as more functionality is added to the libraries, we need to make sure the tutorials reflect the current best practices and are up to date.

Example - a new libtalloc release

https://talloc.samba.org/ points straight to Doxygen docs

NEWS file has the latest entry from 2007..

git log requires knowing where to look in the Samba tree

diff -Naur is not ideal..

If we take a look at talloc, the homepage takes us straight to Doxygen documentation. There are no release notes or changelog. In the tarball, there is a NEWS file, but the last record is from 2007. It's actually required to look into the git history to find out what changed, but even that it not as easy as it could be, because you need to know where the libraries are in the first place. Ideally, navigating to the talloc/tevent homepage should list all the information that developers and maintainers need.

Example - cwrap.org libraries

The libraries were split from Samba and are merged back

Very easy for an outsider to consume

Separate homepage with news and documentation

Development history can be viewed in a separate tree

Several tutorials around the web

After we checked out talloc, we can compare that with another piece of code originating from Samba as well, the cwrap.org libraries. Even though the cwrap libraries live in a separate git tree now, they've been merged back to Samba..I guess that's how Samba rolls.. In comparison with talloc and tevent, the cwrap libraries are much easier for an outside consumer like SSSD to use. The cwrap.org site has a NEWS section and each tarball has a changelog file that lists what has changed since last time. If that is not descriptive enough, it's easy to view the change history for every wrapper separately. For beginners, there is a lwn.net article and other tutorials on the Internet. I'm a bit biased here, because I've been working on the cwrap project with Andreas, mostly on the resolv_wrapper, but I looked at the code and the libraries even before, maybe 2 years ago. But until the project was more separate, the libraries were just too hard to use. For me, cwrap is a role model in how the Samba libraries should be. The difference is not in the code, it's how the code is presented and maintained.

The Samba libraries rock

The libraries Samba produces are already great

With a little more polish, they could be awesome

Let's discuss how we can work together

I've been ranting about Samba libraries for a couple of slides now. I really don't want it to look like I'm not happy with the libraries -- I am. SSSD wouldn't be possible without the libraries that we consume and both talloc and tevent are libraries that more projects should be using. I just think that with just a little more effort on polish and paying attention to how the code is consumed and distributed, the libraries could be used by many more projects. Especially talloc could be used more, memory management is something every C project struggles with. I have a couple of suggestions and wishes that I'd like to present and discuss. Maybe not all of them are realistic or would be welcome by Samba, but let's take a look.

We could do better!

What's new in a release

git-shortlog or a digest to samba-technical is perfectly fine

Keep the documentation up-to-date

The biggest help for us would be to know what are the changes when a new release comes out. We don't need a detailed list describing every commit, but a several bullet points listing the fixes or new API calls would be nice. Maybe this is something we could help with -- send an e-mail to the samba-technical list a day before the release and we can craft a simple changelog file and check if any of the changes should be reflected in the documentation of the libraries. I know this work is not required for the core Samba server, you know your libraries well, so I realize this is a part where we need to help. Feel free to find me after the talk to discuss how, or just write me an e-mail.

We could do even better!

Make it easier to track the individual changes

Decouple libraries from the core Samba tree

Make it clear which interfaces are stable

Do we need to rebuild the libldb memberof plugin with each ldb release?

Let the world know about your libraries!

If I can ask for even more, the best for us would be to be able to see all the changes. I understand it's not required in Samba to send all patches to the mailing list for review and it probably makes sense given the volume of changes in Samba. But from time to time, there are discussions on samba-technical about starting using a code-review tool like Gerrit. Maybe enabling external consumers to track changes and participate in changes to the libraries could be another point for adding the code review tool. From my point of view, and I guess and don't know the historical reasons but for us, it would be best if there was a separate git tree with the libraries or git submodule. I understand that it's a bit sensitive topic, but it would definitely help make the tools better consumable for outsiders and maybe even attract more developers and users. Last but not least, advertise your work! The libraries are great, more people should know about them.

Access control using Group Policy Objects

Access control with SSSD

SSSD has several options for access control

Allow everything
Allow users that match a filter
Allow users present in an ACL
Use Active Directory GPOs

We already have a few ways to control access control in SSSD. The easiest access control is just to allow everything :-). It's possible to configure an LDAP-like filter that allows users who match the filter. Or it's possible to set a list of usernames or groupnames that are allowed. But all these methods are client-side only, in enterprise environment you normally want to define the access control policies at one place. And if SSSD is acting as a Windows domain member, it's most natural to use the GPOs for access control, just like you would with a Windows client machine. This is a feature we added to sssd 1.12 and we used a samba library, libsmbclient to accomplish this feature.

Using GPOs for access control

Group Policy can only be used for access control with SSSD

The rest is not SSSD's domain

But the code is a bit extensible (Sudo?)

Configuring GPO for access control (In Windows)

Create-and-link the GPO in the "Group Policy Management Console"

Configure "User Rights assignments" in the "Group Policy Management Editor"

This is all Windows-specific terminology

LogonRights to PAM services translation

Windows uses LogonRights for access control

Linux uses PAM services

SSSD has default mappings and allows custom mappings in configuration

So the linux client SSSD runs at needs to do translation from the Windows-style aceess control that is done using LogonRights to the Linux-style access control that is done using PAM services. To do that, SSSD has several default mappings of Windows LogonRights and allows the configuration to be changed and overriden. The default mappins are supposed to be useful for most environments -- for example the RemoteInteractiveLogonRight maps onto SSH and InteractiveLogonRight maps onto services like GDM or KDM. There's more mappins, all are listed in the sssd-ad man page. With this mapping, the logon rights configured earlier on the server are translated into client-side access control rules. This is how the GPO access control feature works from user perspective, let's take a look how they are coded up.

Under the hood

Three-step processing

LDAP processing determines what GPOs are applicable
SMB processing fetches the INI GPO policy files
Enforcement parses the INI GPO policy files and does the policy decision

GPO: LDAP processing

Determines which GPOs are applicable to the target computer account

Linked to this OU, enabled, enforced

Retrieves relevant attributes of applicable GPOs

File-system paths for example

Checks if the GPO needs to be fetched from the AD/SMB server

Timeout prevents going to SMB too often

GPO: SMB processing

Fetch the GPO file using libsmbclient

smbc_getFunctionOpen(), smbc_getFunctionRead(), ..

Stores on disk in INI format

Metadata about the INI file in LDB cache

GPO: INI file example


[Unicode]
Unicode=yes
[Version]
signature="$CHICAGO$"
Revision=1
[Privilege Rights]
SeRemoteInteractiveLogonRight = tuser,administrator

GPO: Policy enforcement

Parses the INI files stored previously in GPO cache

Extracts the records correspinging to the Logon Right mapped to the PAM service name

Allows or denies access

Extending LDB with an MDB back end

Why do the work?

A very frequent complaint about SSSD is "it's slow"

SSSD uses its cache heavily due to its architecture

In some cases (saving large groups), the cache was taking a lot of time

Why did I start all this work? One complaint we're getting a lot from our users is that SSSD is slow. When we started looking into the performance results, we found out that writing entries to the database was sometimes the reason for the slowness. Even though some use-cases could be improved by refactoring SSSD code, for instance to perform fewer cache writes, the database performance was still not optimal. Around that time, I heard the idea of using LMDB as an underlying store for LDB. It seemed like it could help our problem, but also seemed like a fun project to hack on!

LMDB 101

LMDB is a very fast, compact key-value store

Written by Howard Chu (Symas/OpenLDAP)

The API usage resembles Berkeley DB

See any of Howard Chu's presentations to learn more (FOSDEM, LDAPCon, ...)

At the same time, LMDB was being introduced on many conferences, blogs and elsewhere. LMDB was written for OpenLDAP by Howard Chu and he made a really good job making sure that the developer community was aware of his work. The advantages of LMDB are performance (L stands for Lightning), while retaining ACID semantics and compact codebase. API-wise, LMDB is close to Berkeley DB. Given that the main driver for starting the work was performance, it's prudent to take a look at some benchmarks to see if it makes sense to implement the LMDB back end.

LMDB benchmarks

Benchmarks comparing different DB engines are available at http://symas.com/mdb/

Pretty graphs!

LMDB wins pretty much all the time

LevelDB's db_bench

https://github.com/hyc/leveldb/tree/benches

Test	LMDB op/s	TDB op/s
fillrandbatch	774503	189630
fillseqsync	129	41

These are the results where TDB comes the closest..

Database benchmarks - tdbench

~~Attend Ralf Boehme's "tbench, a db microbenchmark, or tdb vs all the rest"~~

~~Today, 12:15 PM - 13:00 PM Track 1~~

Database benchmarks - tdbench


./bin/tbench -d lmdb -t 10
Testing with 10 processes, loading 10000 rec each, txn_size: 10
starting bench: load
10 processes created 100000 records in 0.296 seconds (337820 rec/s)


./bin/tbench -d tdb -t 10
Testing with 10 processes, loading 10000 rec each, txn_size: 10
starting bench: load
10 processes created 100000 records in 165.263 seconds (605 rec/s)

The implementation of ldb_mdb

Both TDB and LMDB are key-value stores

Much of the work started as s/tdb/mdb/

But lot of the code could be shared

Because both TDB and LMDB are key-value stores, the API is a bit similar. When I started the work, I mostly copied code, refactored it as I went a bit, but it was apparent there is a lot of code duplication between the tdb and lmdb back ends. Also, while there is an SQLite and LDAP back end in the ldb source, too, realistically only TDB is being used. As a result, the TDB back end implements lot of logic that any new back end needs to implement to be usable, but that is not strictly tied to TDB.

Extensible code

Databases are not one-size fits all

LMDB seems nice for server-side read environments, but some environments might prefer FooDB or BarDB

LevelDB anyone?

RocksDB anyone?

Shared code

Goal: Create the ldb_mdb back end on top of generic modules

Secondary goal: Reuse generic LDB functions in the modules. Some LDB functions were created later than the tdb backend

Shared code could be used also the tdb back end

ldb_mod_*.c

Contains the generic logic currently in the TDB back end

Common module logic
Tevent wrappers
Attribute filtering
..and more

Already there is more common code than mdb-specific code

My work in progess splits several generic modules out of the TDB back end into the common directory. For example, any new back end would probably use tevent in the same manner as TDB uses to establish requests with timeouts and error handling. All search requests need a way to filter only the requested attributes for example or all modify operations need the same common logic to create a resulting ldb_message after a mod. I split all these into modules and wrote unit tests for these in the process. The module interface might not be ideal and I need your help reviewing them!

ldb_mod_* and ldb_mdb status

Implemented & tested

Tevent-based back end initialization from tdb
LDB operations (add, del, modify, search) using mdb
Common logic was split from the operations

In progress

Indexing
Special attributes (case sensitiveness, seqnum, ..)
Permissive control

Testing ldb_mod_* and ldb_mdb

The generic code must be tested to be useful

New code must be compatible with the TDB back end

API tests, not tool tests

Samba, meet cmocka

LDB module benchmarks

We need benchmars for different workloads

SSSD is single-writer

In my tests the LMDB back end was about twice as fast

Show me the code!

It's a work in progress

https://github.com/jhrozek/samba-ldb-mdb/tree/mdb

That's all!

Questions?

https://github.com/jhrozek/sambaxp2015