The Dynamic Filtering Mindset
Api’s are evolving these day, More of more of control is being shifted from the developer to the user. Take OData for example, Many of you may be familiar with OData, A rest-based protocol for dynamically querying, filtering, and updating data. OData can provide help with several different common challenges a developer may face when writing a versatile API. You might use OData to provide a generic search API from your Resources, Or as a simple way for a user to generate custom reports.
Today, I am going to show you how to write a simple api you can use for dynamic filter, but before I do let’s go over some of the benefit and downsides of dynamic filter so you can decide if it’s right for you.
Most of you are already aware of the upsides of dynamic filter. How it provides a way for the client to query data without excessive programming from the developers. But what are the downsides? The downside are there is a decent amount of configuration involved. You need to specify what should be returned to the user, what should be masked or modified. For instance if someone was to request a user you might not want to send the user’s password back. Additionally, dynamic filtering will be slower that explicitly filtering, because you require the api to generate the filters at runtime.
If you are thinking of writing your own dynamic filtering, you should really ask yourself this question before you do. Are the limitation of existing technologies (OData, Dynamic Linq) holding me back from providing the dynamic filtering functionality in my requirements? and is worth it to support your own dynamic filter, which will require extensive testing? With that warning in place, this tutorial for educational purposes of how to write your own simple dynamic filter.
If you would like to follow just use the code to follow long I’ve provided a repository here:
https://github.com/bdb-opensource/Dynamic-Filtering-Tutorial
In this tutorial we will be created a way to dynamically filter Object Properties, data using ‘And’ or ‘Or’ Operations. Although, we are filtering Linq to Objects, I’ve written this in a way in which the generated expressions will also be able to be work with Entity Framework.
Open visual studio and start off by creating a new Console application
Open up your program.cs class.
Let’s create a model to use as test data in which we will filter on.
public class TestData
{
public int Length { get; set; }
public string Name { get; set; }
public string Ethnicity { get; set; }
}
In this model we’ve created with a few properties we can filter on. Essentially, you can modify this to work again any types including complex ones, but here were are only going to cover some of the basic type int, string, bool etc… Note we’re using properties to filter on here for a few reasons.
- You should almost always be using properties they abstract the implementation
- Model binder only will bind on properties
If you want this to work with fields you’ll have to put in a little extra work.
Now that we have something to filter on, let’s create a Model which contains metadata of how to construct our filters.
Lets Define our Enums:
public enum FilterCondition
{
Equal,
}
Filter Condition tells us what condition were using to filter. For now I’ve only provided equals, but you can easily add others such. as GreaterThan or LessThan values, but then you will have to be crafty and explicitly handle use cases for string and bool etc.
Next let’s create a class which will handle whitelisting the properties which we deem filterable. This will prevent the user from filter on things we don’t allow,
public class EntityMetadataService<T>
{
Dictionary<string, Expression<Func<T, object>>> _whiteList;
public EntityMetadataService(Dictionary<string,
Expression<Func<T, object>>> whiteList)
{
this._whiteList = whiteList;
}
public Expression<Func<T, object>> GetMemberExpression(string memberName)
{
Expression<Func<T, object>> memberExpression;
if (false == this._whiteList.TryGetValue(memberName,
out memberExpression))
{
var message = $"Error no filterable property called {memberName}";
throw new ArgumentException(message);
}
return memberExpression;
}
}
Note: The whitelist dictionary in our entity metadata service uses Expression<Func<T, object>>,
This is done so we can store any type of property. When storing a property that points to an Int (which is a value type), this will implicitly add a convert to object which we will later need to strip out.
Next we need a interface with a function to resolve our filters. we’ll take in our metadataService so we know what we can filter on.
public interface IFilterResolver
{
Expression<Func<T, bool>> ResolveFilter<T>(
EntityMetadataService<T> metadataService);
}
Now, Let’s make a concrete class which will handle simple filters. We have not Implemented the `BuildBinaryExpression` function yet, so this will temporarily cause compilation issues.
public class FilterParameterDTO : IFilterResolver
{
public string PropertyName { get; set; }
public FilterCondition FilterCondition { get; set; }
public string Value { get; set; }
public Expression<Func<T, bool>> ResolveFilter<T>(
EntityMetadataService<T> metadataService)
{
var memberExpression = metadataService.GetMemberExpression(
this.PropertyName);
return memberExpression.BuildBinaryExpression<T>(
this.FilterCondition,
this.Value);
}
}
In order to create the BuildBinaryExpression Method we will need to define a few extensions for convenience.
Let’s create a new static class Called ExpressionExtensions:
If you remember before, our whitelist dictionary took in Expression<Func<T, object>> which caused casts to occur for value types. Let’s define a method to strip out the convert expression.
public static class ExpressionExtensions
{
public static LambdaExpression StripConvert( this LambdaExpression expression)
{
if (ExpressionType.Convert != expression.Body.NodeType)
{
return expression;
}
return Expression.Lambda(((UnaryExpression)expression.Body).Operand,
expression.Parameters);
}
}
Let’s also define a method to convert a member expression back to it’s PropertyInfo
public static PropertyInfo ToPropertyInfo( this MemberExpression memberExpression)
{
return (PropertyInfo)memberExpression.Member;
}
To Combine our Expressions with our values, we will need to convert our Filter Condition Enum, to a delegate which represents its action.
let’s create another extension class for enums to handle this.
public static class EnumExtensions
{
public static Func<Expression, Expression, BinaryExpression>
GetBinaryExpressionBuilder(this FilterCondition filterCondition)
{
switch (filterCondition)
{
case FilterCondition.Equal:
return Expression.Equal;
default:
throw new ArgumentException(
$"Filter Condition {filterCondition} is not resolvable");
}
}
}
Next create a Type Extension file for the rest of the convenience methods:
public static class TypeExtensions
{
public static bool IsAssignableTo(this Type type, object obj)
{
if (object.ReferenceEquals(obj, null))
{
return false == type.IsValueType;
}
return type.IsAssignableFrom(obj.GetType());
}
public static Func<string, object> TryGetParseMethod(Type type)
{
if (typeof(string) == type)
{
return (string value) => value;
}
if (type.IsEnum)
{
return (string value) => Enum.Parse(type, value);
}
else if (typeof(Guid) == type)
{
return (string value) => Guid.Parse(value);
}
else if (typeof(Uri) == type)
{
return (string value) => new Uri(value);
}
var methodParse = type.GetMethod("Parse",
new Type[] { typeof(string) });
if (null == methodParse)
{
return null;
}
return (string value) =>
{
if (type.IsAssignableTo(value))
{
return Convert.ChangeType(value, type);
}
return methodParse.Invoke(null, new object[] { value });
};
}
public static Func<string, object> GetParseMethod(Type type)
{
var parseMethod = TryGetParseMethod(type);
if (null == parseMethod)
{
throw new Exception("No Method Parse");
}
return parseMethod;
}
public static object Parse(this Type type, string value)
{
var parseMethod = GetParseMethod(type);
return parseMethod(value);
}
public static bool IsNullable(this Type type)
{
return ((type.IsGenericType) &&
(typeof(Nullable<>) == type.GetGenericTypeDefinition()));
}
}
We don’t need to go over these as much, this is basically a glorified hack to use Duck-Typing on Value Types, to figure out how to parse the values passed to their expected types using reflection.
Navigate back to expression Extensions this is where we’re going to implement the core of our filtering logic. Let’s start by defining out BuildBinaryExpression Method
public static Expression<Func<T, bool>> BuildBinaryExpression<T>(
this Expression<Func<T, object>> expression,
FilterCondition filterCondition, string value)
{
MemberExpression memberExpression = (expression.StripConvert().Body
as MemberExpression);
var propertyInfo = memberExpression.ToPropertyInfo();
var resultType = propertyInfo.PropertyType;
var constValue = resultType.Parse(value);
var constant = Expression.Constant(constValue, resultType);
var parameter = Expression.Parameter(typeof(T));
var property = Expression.Property(parameter, propertyInfo);
var binaryExpressionBuilder = filterCondition.GetBinaryExpressionBuilder();
var binaryExpression = binaryExpressionBuilder(property, constant);
return Expression.Lambda<Func<T, bool>>(binaryExpression, parameter);
}
Let’s explain this:
- We start out by getting a member expression, Which is the expression stored in the whitelist. We need to get the property in which that expression is representing. Since our dictionary it uses expression of T to Object we need to strip out the convert node so our properties that are value types won’t fail
- We will parse our value to the expected type of the property, this uses our parse extensions which rely’s on Duck-Typing
- Create a constant expression which represents our value.
- Create a new parameter for the Expression we’re about to generate
- *Recreate the member expression with the new parameter
- Convert the Filter Expression to the BinaryExpression Delegate
- Build the BinaryExpression
- Cast it back to the return type expected.
*Note: We want to create the expression using a new parameter because in the second part of this tutorial, we will discover that reusing parameter can cause issues. In our case it shouldn’t cause an issue but it is always wise practice to avoid potential ones, if there is not too much overhead.
With this, the foundation for simple filtering is done. Let’s navigate back to Program.cs to write a simple example testing our code, then we will return to expand this to work with logical operators
//Note: we do not allow filtering on ethinicity here to test our whitelist
static Dictionary<string, Expression<Func<TestData, object>>> testDataWhiteList
= new Dictionary<string, Expression<Func<TestData, object>>>
{
[nameof(TestData.Length)] = x => x.Length,
[nameof(TestData.Name)] = x => x.Name,
};
static void Main(string[] args)
{
var test = new TestData {Length = 3, Name = "Win", Ethnicity = "Human"};
var test1 = new TestData {Length = 4, Name = "Lose", Ethnicity = "Human"};
var test2 = new TestData {Length = 5, Name = "Test", Ethnicity = "Human"};
var test3 = new TestData {Length = 6, Name = "Foo", Ethnicity = "Human"};
var filterableData = new List<TestData> { test, test1, test2, test3 };
var simpleFilter = new FilterParameterDTO
{
PropertyName = nameof(TestData.Name),
FilterCondition = FilterCondition.Equal,
Value = "Win"
};
var entityMetadataService = new EntityMetadataService<TestData>(
testDataWhiteList);
var expressionFilter = simpleFilter.ResolveFilter(entityMetadataService);
var results = filterableData.AsQueryable().Where(expressionFilter).ToList();
Console.WriteLine($"Number of Results {results.Count}");
Console.WriteLine($"Is Expected Result: { results.First().Equals(test)}");
Console.ReadLine();
}
In this example we dynamically filter on where the name condition equals win. As you can see we are only return a single result whose name is “Win” as expected.
Next let’s add support for logical operators. First we’re going to need a few things.
Let’s define out enums and the delegates the represent:
public enum ExpressionCombine
{
And,
Or
}
Navigate to the Enum extensions so we can define ExpressionCombineDelegate.
public static Func<Expression, Expression, BinaryExpression>
GetCombineExpressionDelegate(this ExpressionCombine combinor)
{
switch (combinor)
{
case ExpressionCombine.And:
return Expression.And;
case ExpressionCombine.Or:
return Expression.Or;
default:
throw new NotImplementedException();
}
}
This uses the same pattern as the BinaryExpression Builder to translate our enums to delegates.
Let’s create a new for a new FilterResolver which will be able to handle combinations of any IFilterResolver.
public class BinaryExpressionDTO : IFilterResolver
{
public IFilterResolver Left { get; set; }
public IFilterResolver Right { get; set; }
public ExpressionCombine Combinor { get; set; }
public Expression<Func<T, bool>> ResolveFilter<T>(
EntityMetadataService<T> metadataService)
{
var leftExpression = this.Left.ResolveFilter(metadataService);
var rightExpression = this.Right.ResolveFilter(metadataService);
return leftExpression.Combine(rightExpression, this.Combinor);
}
}
Note: Just like last time we will leave the Combine Extension Method is not defined yet so we can define some Convenience Methods first.
Resolve filter is called recursively to resolve the filter whether it’s a binary expression or a FilterParameter.
public class ReplaceVisitor : ExpressionVisitor
{
private readonly Expression _from;
private readonly Expression _to;
public ReplaceVisitor(Expression from, Expression to)
{
this._from = from;
this._to = to;
}
public override Expression Visit(Expression node)
{
return node == this._from ? this._to : base.Visit(node);
}
}
This replace visitor takes care of combining 2 simple expressions parameters. Take for instance these two expression x => x.Id == 3, and x => x.name == “Brian”. Notice that the x is defined in both expressions. If were were to attempt to combine these two expressions, with a logical `Or` using their bodies, it would generate something like this x => x.Id == 3 || x.Name == “Brian”.
Even though this looks correct it is NOT.
In this expression the x.Id points to the original parameter where it was defined. Even though the variables names are the Same they actually are not the same Variable. This is something that is important to know when dealing with an expression.
Let’s define a method to replace parameters, inside of our expression extensions, which will use the ReplaceVisitor:
public static Expression<Func<T1, T2>> ReplaceParameter<T1, T2>(
this Expression<Func<T1, T2>> expression, ParameterExpression param)
{
var replaceVisitor =
new ReplaceVisitor(expression.Parameters.First(), param);
var replacedExpression = replaceVisitor.Visit(expression.Body);
return Expression.Lambda<Func<T1, T2>>(replacedExpression, param);
}
Just to reiterate this method replaces the input parameter with the one supplied instead.
Now we can define our Combine Method:
public static Expression<Func<T1, T2>> Combine<T1, T2>(
this Expression<Func<T1, T2>> first,
Expression<Func<T1, T2>> second,
ExpressionCombine combinor)
{
var param = Expression.Parameter(typeof(T1), "x");
var newFirst = first.ReplaceParameter(param);
var newSecond = second.ReplaceParameter(param);
var expressionCombinor = combinor.GetCombineExpressionDelegate();
var combinedExpression = expressionCombinor(newFirst.Body,
newSecond.Body);
return Expression.Lambda<Func<T1, T2>>(combinedExpression, param);
}
1. Create a new parameter to replace the input parameters in the two expressions we want to combine,
Note: Both Expressions are of T1 meaning they use the same input type of parameter. This allows us to replace both expression’s parameter with the one we define.
2. We replace the parameter reference in both expressions with the new one we created.
3. Use the Combine Enum to resolve the expression Delegate which combines the expression.
4. Combine the expressions together.
5. We cast our expression back to the type Expected Type.
Let’s navigate back to our Program.cs a simple example to verify our filtering works. Add in this code before the Console.ReadLine() :
var simpleFilter2 = new FilterParameterDTO
{
PropertyName = nameof(TestData.Length),
FilterCondition = FilterCondition.Equal,
Value = "6"
};
var binaryExpression = new BinaryExpressionDTO { Left = simpleFilter,
Right = simpleFilter2,
Combinor = ExpressionCombine.Or };
var binaryExpressionFilter = binaryExpression.ResolveFilter(
entityMetadataService);
results = filterableData.AsQueryable()
.Where(binaryExpressionFilter)
.ToList();
Console.WriteLine($"Number of Results {results.Count}");
Console.WriteLine(
$"Is Expected First Result { results.First().Equals(test)}");
Console.WriteLine(
$"Is Expected Second Result {results.Skip(1).First().Equals(test3)}");
If your run the program you can see that our filter is working properly. I’ve provided a sample repo on where you can see the full project
This should serve as a good resource understanding how to filter dynamically. Feel free to modify the design to Support Fields, Other Logical Operator or Filter Conditions.
Stay Tuned for the Next tutorial where I will go over how to model bind to interfaces, so we can use an IFilterResolver in our Controllers.