markrendle / simple.data

A light-weight, dynamic data access component for C# 4.0

Simple.Data

A lightweight, dynamic data access component for .NET, written in C#.

What is it?

Prompted by the need for an easy-to-use database access component which prevents SQL injection attacks while not requiring lots of boilerplate ADO.NET code or a pre-generated ORM model. Inspired by Ruby's ActiveRecord and DataMapper gems.

Instead of

public User FindUserByEmail(string email)
{
	User user = null;
    using (var connection = new SqlConnection(ConfigurationManager.ConnectionStrings["Default"].ConnectionString))
    using (var command = new SqlCommand("select id, email, hashed_password, salt from users where email = @email", connection))
    {
        command.Parameters.Add("@email", SqlDbType.NVarChar, 50).Value = form.Email);
	    connection.Open();
	    using (var reader = command.ExecuteReader())
	    {
		    if (reader.Read())
		    {
			    user = new User {Id = reader.GetInt32(0), Email = reader.GetString(1), Password = reader.GetString(2), salt = reader.GetString(3)};
			}
		}
	}
	return user;
}

why not just write

public User FindUserByEmail(string email)
{
	return Database.Open().Users.FindAllByEmail(email).FirstOrDefault();
}

and take the rest of the morning off?

Simple.Data does this by using the dynamic features of .NET 4 to interpret method and property names at runtime and map them to your underlying data-store with a convention-based approach. For the code above, there is no pre-defined type with a Users property, and no FindByEmail method. Within Simple.Data, that single line of code is converted into all the ADO.NET boilerplate for you.

Multiple database and NoSQL store support

Because Simple.Data provides a sort of dynamic Domain Specific Language to represent queries, inserts, updates and deletes, it is able to support not only a wide range of RDBMS engines, but also non-SQL-based data stores such as MongoDB. It has an open and flexible model of Adapters and Providers which make it simple to write plug-ins to map to almost any back-end.

Currently, Simple.Data has adapters for:

• ADO-based access to relational databases, with providers for:
  • SQL Server 2005 and later (including SQL Azure)
  • SQL Server Compact Edition 4.0
  • Oracle
  • MySQL 4.0 and later
  • SQLite
  • PostgreSQL
  • SQLAnywhere
  • Informix
• MongoDB
• OData

Work is in progress to support Azure Table Storage. I'm also ensuring that Simple.Data works on Mono by the 1.0 release.

If you'd like to create an adapter or provider and need some help to get started, drop in on the mailing list (see below).

Resources

• Simple.Data can be installed from NuGet
• Find more information in the wiki
• Documentation! (and here is the SimpleFx Github Project if you want to help us improve the documentation)
• Ask questions or report issues on the mailing list
• Follow @markrendle on Twitter for updates
• Check out my blog for the latest news

simple.data's Issues

Caching layer?

Great project!! Any plans for a caching layer? LIke NHibernate's second level cache?
I guess this would only be appropriate for SQL back ends.

Can't Update -- Even using your Example

I'm attempting to update using your example code:

  var customer = db.Customers.FindByCustomerId(1);
  customer.Address = "Changed";
  db.Customers.Update(customer);

I just pulled the latest and am running against SQL Server DB. It's throwing an "Specified Method is not Supported" exception:

Simple.Data\Simple.Data.Ado\OptimizedDictionary`2.cs Line: 202

set { throw new NotSupportedException(); }

Thanks.

Be more tolerant of what properties DTOs have defined

I got a call today from one of our tems that are using Simple.Data. He said that he had a special property on his DTO and SiD would try to persist the column and complain that it doesn't exist.

I was wondering - what if the column set drives what gets persisted and any extra properties on a DTO gets ignored?

Feature Request: Narrow down the SELECT columns.

Is that possible now? If not, could it be added?

Thanks!

Issue when persisting an instance

I have defined this entity:

public class Post
{
  public int Id { get; protected set; }
  public string Slug { get; set; }
  public string Text { get; set; }
}

If I insert into the database with db.Posts.Insert(Slug: "foo", Text: "Bar") is everything handled correct. If I on the other hand create new Post { Slug = "foo", Text = "bar" } and passes it to the method, the slug column in the database is set to 0, and the text column is set to foo. The bar value is not persisted anywhere.

Issue with Guids?

Executing this query "db.Ratings.FindAllByUserAccountId(userAccount.Id);"

gives me
System.InvalidCastException : Invalid cast from 'System.String' to 'System.Guid'.

"userAccount.Id" is a Guid and in the database a uniqueidentifier

Please Publish TestHelper to NuGet

Hi Mark,

I finally got around to updating my provider to support the latest version of Simple.Data. I had to hand build the TestHelper lib in order to use the extensions you created to give me access to the connection. It would be nice if you published this to nuget as well.

Thanks,
Bobby

Feature Request: Order By

I'm sure it's planned... but nice to have order by available.

Add Update method

There should be a plain Update method which uses the Primary Key as criteria.

Updating a binary causes a CastException

Updating a binary in MS-SQL seems to fail with a cast exception. I've tried this with Content set both to type image and varbinary(max).

using System.IO;
using NUnit.Framework;

namespace Simple.Data.ByteBug {
    [TestFixture]
    public class ByteBug {
        private const string connectionString = "server=.;database=simpleDataTest;uid=testuser;pwd=testpassword;";

        /*
         * --SQL definition of table Logo
         * CREATE TABLE dbo.Logo (SiteId int NOT NULL, Name nvarchar(50) NOT NULL, MimeType nvarchar(50) NOT NULL, Content] image NOT NULL)  ON [PRIMARY] TEXTIMAGE_ON [PRIMARY]
         * GO
         * ALTER TABLE dbo.Logo ADD CONSTRAINT PK_Logo PRIMARY KEY CLUSTERED  (SiteId) WITH(STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
         * GO 
         * ALTER TABLE dbo.Logo SET (LOCK_ESCALATION = TABLE)
         * GO
         */

        [Test(Description = "Updating an existing logo is possible")]
        public void Save_AnExistingLogo_Possible() {
            const string fileName1 = "document-new.png"; //doesn't matter which - these were from the Tango project
            const string fileName2 = "document-open.png";

            var bytes = getBytes(fileName1);

            var itemToSave = new Logo {
                MimeType = "image/png",
                Name = "testlogo.png",
                SiteId = 1,
                Content = bytes
            };

            var database = Database.OpenConnection(connectionString);

            database.Logo.DeleteBySiteId(1); //clean-up, if necessary

            database.Logo.Insert(itemToSave);

            var persisted = database.Logo.FindBySiteId(1);

            persisted.Content = getBytes(fileName2);

            database.Logo.Update(persisted);

            /*
             * failed: System.InvalidCastException : Failed to convert parameter value from a Byte to a Byte[].
             * ----> System.InvalidCastException : Invalid cast from 'System.Byte' to 'System.Byte[]'.
             * at System.Data.SqlClient.SqlParameter.CoerceValue(Object value, MetaType destinationType)
             * at System.Data.SqlClient.SqlParameter.GetCoercedValue()
             * at System.Data.SqlClient.SqlParameter.Validate(Int32 index, Boolean isCommandProc)
             * at System.Data.SqlClient.SqlCommand.BuildParamList(TdsParser parser, SqlParameterCollection parameters)
             * at System.Data.SqlClient.SqlCommand.BuildExecuteSql(CommandBehavior behavior, String commandText, SqlParameterCollection parameters, _SqlRPC& rpc)
             * at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async)
             * at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result)
             * at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(DbAsyncResult result, String methodName, Boolean sendToPipe)
             * at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
             * at Simple.Data.Ado.AdoAdapter.TryExecute(IDbCommand command)
             * at Simple.Data.Ado.AdoAdapter.Execute(ICommandBuilder commandBuilder)
             * at Simple.Data.Ado.AdoAdapter.Update(String tableName, IDictionary`2 data, SimpleExpression criteria)
             * at Simple.Data.Database.Update(String tableName, IDictionary`2 data, SimpleExpression criteria)
             * at Simple.Data.Commands.UpdateCommand.UpdateByKeyFields(String tableName, DataStrategy dataStrategy, Object entity, IEnumerable`1 keyFieldNames)
             * at Simple.Data.Commands.UpdateCommand.Execute(DataStrategy dataStrategy, DynamicTable table, InvokeMemberBinder binder, Object[] args)
             * at Simple.Data.DynamicTable.TryInvokeMember(InvokeMemberBinder binder, Object[] args, Object& result)
             * at CallSite.Target(Closure , CallSite , Object , Object )
             * at System.Dynamic.UpdateDelegates.UpdateAndExecuteVoid2[T0,T1](CallSite site, T0 arg0, T1 arg1)
             * ByteBug.cs(43,0): at Simple.Data.ByteBug.ByteBug.Save_AnExistingLogo_Possible()
             * --InvalidCastException
             * at System.Convert.DefaultToType(IConvertible value, Type targetType, IFormatProvider provider)
             * at System.Byte.System.IConvertible.ToType(Type type, IFormatProvider provider)
             * at System.Convert.ChangeType(Object value, Type conversionType, IFormatProvider provider)
             * at System.Data.SqlClient.SqlParameter.CoerceValue(Object value, MetaType destinationType)
             */
        }

        private byte[] getBytes(string fileName) {
            var file = new FileInfo(fileName);
            var fileLength = file.Length;
            var imageStream = new FileStream(file.Name, FileMode.Open, FileAccess.Read);
            return new BinaryReader(imageStream).ReadBytes((int)fileLength);
        }
    }

    public class Logo {
        public string MimeType { get; set; }
        public string Name { get; set; }
        public int SiteId { get; set; }
        public byte[] Content { get; set; }
    }
}

NotImplementedException in InsertCommand.cs when in a transaction

I am getting a null reference exception as a result of hitting this NotImplementedException, is this work in progress?

InvalidOperationException when running LIKE query after another

This works:
var db = SD.Database.OpenFile(Server.MapPath("~/App_Data/Northwind.sdf"));
var data = db.Products.FindAll(db.Products.ProductName.Like("C%"));
var products = db.Products.FindAll(db.Products.CategoryId == 4);

Whereas this throws an InvalidOperationException (Operation is not valid due to the current state of the object):
var db = SD.Database.OpenFile(Server.MapPath("~/App_Data/Northwind.sdf"));
var products = db.Products.FindAll(db.Products.CategoryId == 4);
var data = db.Products.FindAll(db.Products.ProductName.Like("C%"));

all this internal stuff is making me cry

is there a simple way to fake a database and validate the sql generated? Similar to what you do in the Database Behavoir tests?

Between statement

Version 0.45 seems to be incorrect when it comes to between statements as a result of the Range extension.

In Simple.Data.Ado.ExpressionFormatter the FormatRange will output parantheses around the values, which will cause an execption when the sql statement is executed.

Verified on Sql Server express 2008, SQL server CE 4.0 and Sql server 2005.

FindAll

Is FindAll broken? FindAllBy works fine, and I can probably just create a FindAllById(db.myTable.Id > -1) or something like that.

Getting a "Query returned multiple rows; Cannot return scalar value." after doing "Take(1)"

When I run this code when no previous photo should be found, I get an exception saying "Query returned multiple rows; Cannot return scalar value.". This is strange because it shouldn't return any values and because I use ".Take(1)" to make sure only one row is returned.

string previousSlug = DB.Photos
                        .Query()
                        .Select(DB.Photos.Slug)
                        .Where(DB.Photos.Published == true && DB.Photos.DatePublished < photo.DatePublished.Value) // There is no published photo with a smaller DatePublished than the one I check against
                        .OrderByDatePublishedDescending()
                        .Take(1) // Should return only one record no matter what
                        .ToScalarOrDefault<string>();

Feature Request: Add Generation Logging/Debugging Options

Insert isn't working for me. Not sure why. Had to put some Debug.WriteLine() type of code in the command execute just to see what kind of code it was generating. I realize that Mock Database does that, but another option for when you need to see what's being run would be helpful.

Thanks.

Nullable fields in XmlMockAdapter

How would one specify a nullable field and put a null value into that field in the xml?

Inherited interfaces not working correctly

Not sure if this is intended, but if I have a class that is composed of inherited interfaces, the base interface properties do not get surfaced in an All retrieval. Instead of getting back an instance of clazz property populated, I get back a KeyValuePair<string,object> that contains {InstanceId : value of instanceid}.

public interface IBaseEntity
{
public virtual Guid Id { get; set; }
}

public abstract class BaseEntity : IBaseEntity
{
public virtual Guid Id { get; set; }
}

public interface IInstanceDefinedBaseEntity : IBaseEntity
{
Guid InstanceId { get; set; }
}

public abstract class InstanceDefinedBaseEntity : BaseEntity, IInstanceDefinedBaseEntity
{
public Guid InstanceId { get; set; }
}

public class clazz: InstanceDefinedBaseEntity
{

}

Column aggregates

Min,Max,Average,Sum

For the love of god how do I get at the connection?

Hi Mark,

I have been trying to get my SQLite provider updated and have been having little luck. It looks like the ProviderLocator is no longer a static class that I can use to weasel my way into the active connection. You can see here why I need to get at the connection:

https://github.com/NotMyself/Simple.Data/blob/master/Simple.Data.SqliteTests/InMemoryUsageTests.cs#L20

I am basically using the connection to inject some tables and test data into an im memory DB prior to doing some tests. Have any hints or thoughts on a better way to accomplish this?

Remember that if the connection is closed the DB is destroyed and if I open up two connections.... sqlite is kind enough to create two DBs for me.

/stuck in seattle
Bobby

Aliasing

.As(str) on columns and tables

Explicit joins

Join(db.Y, db.X.yid == db.Y.id)

Definition of "Database Profiles", introduction of 2 extension points

Hi Mark,
I would like to put a few thoughts down, see if it's something you like.
I was thinking of how a user could add more "meta"information to some kind of connection and one ide would be to allow connection profiles to be defined. A connection profile would be similar to a named connection. To define a profile one would access the "DatabaseOpener" and call "CreateProfile". The caller needs to specify

• a profile name (or the any option, which would apply any provisions of "stuff" to any connection obtained)
• a connect string, or how to obtain one
• add implementation of existing plugin interfaces.

This introduces a new way to get the dynamic db, namely via DatabaseOpener.Profiles.ProfileName() : dynamic.

Internally, Simple.Data would get a small container which contains any plugin interface implementations that come in via a profile

In order to cover issue #50 , we could introduce two interfaces, one specific to ado, which is allowed to inspect IDbCommand instances created by Simple.Data to interact with a db.

interface IAdoCommandPlugin {
  void Inspect(IDbCommand cmd);
}

The other hook is to inspect and possibly modify the IDictionary<string,object> returned by a call to the data store:

interface IDataStoreReturnPlugin {
  IDictionary<string,object> Inspect(IDictionary<string,object> values);
}

Now, this would allow extension of Simple.Data without pushing more features into the core.

Catching DBnull values during implicit casting

I somehow ended up with a bunch of Null values in my test db, and this caused argument exceptions when I ran my find tests.
The exceptions originated in the ConcreteTypeCreator class TryCreate method.
I found that adding two lines of code fixed this:
(Added :
if (value == dbnull.value)
value = null;

My git foo is a bit rusty, and for some reason when I change anything, Git insists on producing a diff that removes all lines, and then re adds them, so a git commit + pull req would be useless until I figure this out, so pasted code is all I can bring now..

public bool TryCreate(IDictionary<string, object> data, out object result)
{
bool anyPropertiesSet = false;
object obj = Activator.CreateInstance(_concreteType);
object value;
foreach (var propertyInfo in _concreteType.GetProperties().Where(pi => CanSetProperty(pi, data)))
{
value = data[propertyInfo.Name.Homogenize()];

            if (ConcreteCollectionTypeCreator.IsCollectionType(propertyInfo.PropertyType))
            {
                if (!ConcreteCollectionTypeCreator.TryCreate(propertyInfo.PropertyType, (IEnumerable)value, out value))
                    continue;
            }
            else
            {
                var subData = value as IDictionary<string, object>;
                if (subData != null && !ConcreteTypeCreator.Get(propertyInfo.PropertyType).TryCreate(subData, out value))
                    continue;
            }

            if (value == DBNull.Value)
                value = null;

            propertyInfo.SetValue(obj, value, null);
            anyPropertiesSet = true;
        }

        result = anyPropertiesSet ? obj : null;

        return anyPropertiesSet;

Identity fields in XmlMockAdaptor

Is it possible to specify an identity field somehow and have the XMLMockAdaptor return a value in that field as the result of an insert?

Issue with Find, multiple schemas, and duplicate table names

Slightly obscure one here - but it's causing problems against our db.

Two schemas. A table in either schema named ApplicationLog.

An API call to db.dbo.ApplicationLog.Find(db.dbo.ApplicationLog.Message == "Application starting"); blows up with "System.InvalidOperationException : Sequence contains more than one element"
The stacktrace makes it quite clear what is going on:
at System.Linq.Enumerable.SingleOrDefault(IEnumerable`1 source)
at Simple.Data.Ado.Schema.TableCollection.FindTableWithName(String tableName)
instead of calling the overload with a schema in it.

FindAllBy syntax seems to work just fine.

Same issue with updates

My updates are failing as well. Great project by the way. I am a Big Fan.

Nested Object Aliases

Mark,
I was adding alias support into the mongodb adapter and hit a snag. It all works great until I need to alias a nested object and it's descendents. For example, given the structure that users have an address, the following is the syntax I would need.

original shape:

usrs:
    adr:
        cty
        st

select portion of Query()

.Select(db.usrs.adr.As("Address")
    .Select(db.usrs.adr.cty.As("City"), db.usrs.adr.st.As("State"))

aliases shape:

usrs:
    Address:
        City
        State

I was going to add a Select statement onto ObjectReference and let the ObjectReference hold it's childrens selections, but wasn't sure if there were any problems that would cause. If that seems ok, I'll do the work and send a pull request. If you can think of another way, please let me know.

Feature request: ToScalarList<T>

For example:

IEnumerable<DateTime> allDates = DB.Users
                                   .All()
                                   .Select(DB.Users.DateOfBirth)
                                   .ToScalarList<DateTime>();

Grouping

GroupBy(db.X.foo, db.Y.bar)

Improve exception handling

Add generated SQL to exceptions thrown in AdoAdapter

Conversion capabilities between .NET and DB values

Introduce converter functions which you can attach to columns. Syntax-wise it could look something like this:

var converterPlugin = new ConverterPlugin();
converterPlugin.Add(_db.Customers.Id, g => g.ToByteArray(), v => new Guid((byte[])v));
converterPlugin.Add(col=>col.Name.EndsWith("BOOL"), BoolToDb, DbToBool);

DatabaseOpener.Profiles.Add(cfg=> {
  cfg.Name = "ORCL";
  cfg.Connectionstring("...");
  cfg.AddPlugin(converterPlugin);
});

_db = Database.OpenProfile().ORCL;

Converters are applied when searching/updating inserting in that a provided .NET value passes through a converter, or when reading back values in that the DB value is converted to some .NET type.

It should have support for a single column definition, or a definition that defines a predicate that checks whether a given column should get the converters applied. That way one can do conversions for e.g. a specific column data type or based on some column naming convention.

This feature is based on the ideas put down in #56

Feature Request: Bulk Operations

Thanks for all your hard work.

It's probably on your list... but just something like UpdateAll, DeleteAll and InsertAll.

Simple, I know, but cleaner than having to do a foreach over a list. Plus, could possibly bulk the SQL.

Paging

Skip and Take

Pluralisation inference

How about away to switch on pluralisation inference? For example, I like my table names to be singular e.g. "Customer". So my query looks like "db.Customer", I would like "db.Customers"

Specific columns

Select function

Database.Open(connectionName)

An overload of Database.Open which takes a Connection Name as specified in the .config file for the app.

Ordering

OrderBy, OrderByDescending, ThenBy, ThenByDescending

Update with NULL value doesn't update

Setting a property to NULL and then calling Update doesn't set that value to NULL in the database.

Command.Connection always gets disposed when using FindAll, even when using a DelegatedDbConnection

For the SQLite memory database we use a delegatedbconnection. The problem is that the connection gets disposed directly upon closing the IDataReader.

This is the relevant part

// from Simple.Data.Ado.DbCommandExtensions.cs

        private static void DisposeCommandAndReader(IDbCommand command, IDataReader reader)
        {
            // by calling dispose on command.Connection, the delegateddbconnection is skipped
            using (command.Connection)
            using (command)
            using (reader)
            { /* NoOp */ }
        }

Homogenization

One more place it looks like I forgot to kill. ConcreteTypeCreator line 41 needs to have the homogenization removed. This means that whatever dictionary the adapter passes in is how keys will get compared. Ado uses a HomogenizedKeyComparer, while Mongo will use a CaseSensitive one.

Full CRUD in XmlMockAdapter

Enable Insert, Update and Delete operations in XmlMockAdapter.

Possible Bug in ForeignKeyCollection

I just checked in a test that demonstrates a bug that I am currently running into. I have a table with two foreign keys and the ForeignKeyCollection seems to be bonking because the collection uses only the master table ObjectName as a key for the collection. You can see the test here: https://github.com/NotMyself/Simple.Data/blob/master/Simple.Data.Ado.Test/ForeignKeyCollectionTests.cs

Are my assumptions correct? I'll try to submit a patch but I will need to dig a little deeper into how the key collection is used and referenced by the consuming code.

Minor: Your Range Test isn't Localized.

Unit test "RangeTest.cs" fails because we have a little different date style on this side of the pond:

Expected: 2011-01-01 00:00:00.000
But was:  "01/01/2011"

Add multi-entity querying

Joins for RDBMS, sub-doc query for document stores.

Feature Request: Paging

Unless I'm missing something... no paging?

Table aggregate

Count() available in column lists.

Issue: Insert(dynamic) needs more options

In DynamicTable.cs Insert(dynamic) -- it wasn't working with my statically typed object. I modified with the following code:

var dictionary = entity as IDictionary<string, object> ?? ObjectEx.ObjectToDictionary((object)entity);

if (dictionary != null)
{
  _dataStrategy.Insert(_tableName, dictionary);
}

Thanks.

Conversion capabilities between .NET and DB values

Introduce converter functions which you can attach to columns. Syntax-wise it could look something like this:

_db = Database.Open(...);
_db.Settings.Converters.Add(_db.Customers.Id, g => g.ToByteArray(), v => new Guid((byte[])v));
_db.Settings.Converters.Add(col=>col.Name.EndsWith("BOOL"), BoolToDb, DbToBool);

Converters are applied when searching/updating inserting in that a provided .NET value passes through a converter, or when reading back values in that the DB value is converted to some .NET type.

It should have support for a single column definition, or a definition that defines a predicate that checks whether a given column should get the converters applied. That way one can do conversions for e.g. a specific column data type or based on some column naming convention.