Dotnet on Chris McKelt - Remembering Thoughts

Perth .Net user group talk

Wed, 08 Aug 2018 00:00:00 +0800

Predicting BS using ML.Net

Machine learning is typically the realm for R/Python. But can .Net move into this space?

In this talk we will run through Microsoft’s new ML.Net framework including what it currently offers, how to build a learning pipeline and how to deploy a model to an Azure Service.

https://slides.com/chrismckelt/deck-451cdb94-a37d-47b7-9d49-6686065e7d03

Charge Id – The prediction model

Sun, 29 Jul 2018 00:00:00 +0800

In this post we will build a prediction model using ML .Net to classify an individual bank transaction into a pre-defined group using the description text.

We will then evaluate the effectiveness of the model using standard evaluation metrics.

The steps involved to create a model using ML.Net are:

Decide what type of machine learning problem we are trying to solve?
Retrieve and clean the data (CSV/TSV used with ML.Net for no)
Create a prediction service API
Create a ‘Learning Pipeline’ to iterate over the model build process
- Training
- Testing
- Evaluating result metrics
Exploring the output model file

Charge Id – Analysing the data

Sat, 21 Jul 2018 00:00:00 +0800

So what are we trying to do here?

Hypothesis

Given a users bank statement we will be able to predict (within a statistical confidence level ) the transactions within a period into categories & sub categories for spending classification.

Sub categories available on this link

Charge Id – solution overview

Sun, 15 Jul 2018 00:00:00 +0800

The proposed solution sequence flow follows:

Posts in this series

Charge Id – scratching the tech itch
Charge Id – lean canvas
Charge Id – solution overview
Charge Id – analysing the data
Charge Id – the prediction model
Charge Id – deploying a ML.Net Model to Azure

Code

https://github.com/chrismckelt/vita

Charge Id - Lean Canvas

Fri, 13 Jul 2018 00:00:00 +0800

Lean Canvas

Problem

This is the start of a series of blog posts where I try to build an online personal budgeting system.

The motivation for this was me trying to do a household budget usin

With Open Banking approaching a common service to classify income and expenditure for bank statement transactions is not currently available on the market.

The ability for a consumer to categorise their transactions sits within varied personal finance providers (eg. pocketbook, mint).

Charge Id – What is the problem?

Fri, 06 Jul 2018 00:00:00 +0800

This is the start of a series of blog posts where I try to build an online personal budgeting system.

The motivation for this was me trying to do a household budget using existing tools (Pocketbook, CashDesk, QuickBooks, Xero).

Each time I needed to classify a large number of transactions manually (except Pocketbook they have got a pretty good classification engine)

As I do lots of transfer between accounts the summaries never made sense. So I thought Id give it a go to build my own personal finance budgeter.

Map .html extension to the .Net razor view engine

Thu, 10 Sep 2015 00:00:00 +0800

1. Add the buildProvider config for razor inside the compilation element

2. Application start –> Registers the html extension with razor

3. Create a Start.cshtml page and ingest the index.html page

<compilation debug\="true" targetFramework\="4.6" \>
 <buildProviders\>
 <add extension\=".html" type\="System.Web.WebPages.Razor.RazorBuildProvider"/>
</buildProviders\>
</compilation\>

 System.Web.Razor.RazorCodeLanguage.Languages.Add("html", new CSharpRazorCodeLanguage()); WebPageHttpHandler.RegisterExtension("html");


@{

 Layout \= null;

 @RenderPage("~/index.html")

}

<!-- Version + @System.Reflection.Assembly.GetAssembly(typeof (Startup)).GetName().Version.ToString(); \-->

Web API Test to ensure public routes do not change

Mon, 19 May 2014 00:00:00 +0800

[TestFixture]
public class RouteTestFixture {

 [Test]
 public void Routes _should _not _change() {
 const string good = @ "all existing routes go here (hint run the test first then copy output to here)";
 var goodRoutes = good.Split(Convert.ToChar(";"));
 var notFound = new List < string > ();
 var ass = Assembly.GetAssembly(typeof (EventStatisticsController));
 var sb = new StringBuilder();
 foreach(var type in ass.GetTypes()) {
 try {
 var members = type.GetMembers();
 for (int i = 0; i < members.Length; i++) {
 if (members[i].IsDefined(typeof (RouteAttribute), false)) {
 Object[] atts = members[i].GetCustomAttributes(typeof (RouteAttribute), false);
 for (int j = 0; j < atts.Length; j++) {
 var routeAttribute = (RouteAttribute) atts[j];
 string route = routeAttribute.Template + @ "/" + routeAttribute.Name;
 Console.WriteLine(route);
 sb.Append(route + ";");
 if (!goodRoutes.Contains(route)) {
 notFound.Add(route);
 }
 }
 }
 }
 } catch (Exception e) {
 Console.WriteLine(@ "An exception occurred: {0}", e.Message);
 }
 }
 if (notFound.Any()) {
 Console.WriteLine(@ "-- MISSING ROUTES --");
 foreach(var nf in notFound) {
 Console.WriteLine(nf);
 }
 Assert.Fail("Missing Routes: " + notFound.Count);
 } else {
 Assert.True(true, "All routes matched");
 }
 Console.WriteLine(@ "------------------------------------------------");
 Console.WriteLine(@ "Match the following string for future tests");
 Console.WriteLine(sb.ToString());
 }
}

Custom NHibernate User Type

Thu, 26 Aug 2010 00:00:00 +0800

How to use a custom NHibernate User Type

To store the state of a Risk (ie RiskState) as a value in a column we can use a NHibernate.UserType

The RiskState can be one of the following: (these are all derived from RiskState which implements IRiskState)

Our user type

 public class RiskStateNameUserType: IUserType {
#region IUserType Members
public object Assemble(object cached, object owner)
{
return cached;
}
public object DeepCopy(object value)
{
return value;
}
public object Disassemble(object value)
{
return value;
}
public int GetHashCode(object x)
{
return x.GetHashCode();
}
public bool IsMutable
{
get {
return false;
}
}
public object NullSafeGet(System.Data.IDataReader dr, string\[\] names, object owner)
{
var property0 = NHibernateUtil.String.NullSafeGet(dr, names\[0\]);
if (property0 == null)
{
return null;
}
IRiskState state;
if (owner is Risk)
{
state =
(IRiskState)
Activator.CreateInstance(Type.GetType(typeof (IRiskState).Namespace + "." + (string) property0),
owner);
} else
{
state =
(IRiskState)
Activator.CreateInstance(Type.GetType(typeof (IRiskState).Namespace + "." + (string) property0));
}
return state;
}
public void NullSafeSet(System.Data.IDbCommand cmd, object value, int index)
{
if (value == null)
{
((IDataParameter) cmd.Parameters\[index\]).Value = DBNull.Value;
} else
{
var state = (IRiskState) value;
((IDataParameter) cmd.Parameters\[index\]).Value = state.GetType().Name;
}
}
public object Replace(object original, object target, object owner)
{
return original;
}
public Type ReturnedType
{
get {
return typeof (RiskState);
}
}
public NHibernate.SqlTypes.SqlType\[\] SqlTypes
{
get {
return new\ [\] {
NHibernateUtil.String.SqlType
};
}
}
public new bool Equals(object x, object y)
{
if (x == null && y == null) return true;
if (x == null || y == null) return false;
return x.GetType() == y.GetType();
}
#endregion
}

Using this on a property as follows:

WPF UI Thread Dispatcher

Wed, 26 May 2010 00:00:00 +0800

A simple implemention for calling asych methods from the UI

Examples

 dispatcher.ExecuteOnMainUIThread(CommandManager.InvalidateRequerySuggested);
dispatcher.Execute(() =>
{
SomeLongRunningMethodHere();
});

The interface

public interface IDispatcher { void Execute(Action action); void ExecuteOnMainUIThread(Action action); } }

Synchronous for use in Testing

 public class SynchronousDispatcher: IDispatcher {
public void Execute(Action action) {
action();
}
public void ExecuteOnMainUIThread(Action action) {
action();
}

}

Asynchronous for use by the application at run time

public class AsynchronousDispatcher: IDispatcher {
public void Execute(Action action) {
action.BeginInvoke(CallBack, action);
}
public void ExecuteOnMainUIThread(Action action) {
Dispatcher dispatcher;
if (Application.Current != null) {
dispatcher = Application.Current.Dispatcher;
} else {
dispatcher = Dispatcher.CurrentDispatcher;
}
dispatcher.Invoke(action);
}
private void CallBack(IAsyncResult result) {
try {
((Action) result.AsyncState).EndInvoke(result);
} catch (Exception ex) { // Need to raise the exception on the main thread ExecuteOnMainUIThread(() => { throw ex; } ); } finally { result.AsyncWaitHandle.Close(); }
}
}

Stream bytes to files

Wed, 10 Feb 2010 00:00:00 +0800

using(var stream =
 Assembly.GetAssembly(typeof (StubPolicy)).GetManifestResourceStream(
 "Documents.TestHelpers.Files.test.msg")) {
 const int bufferLength = 256;
 var buffer = new Byte\[bufferLength\];
 if (stream != null) {
 int bytesRead = stream.Read(buffer, 0, bufferLength);

 using(var fs = new FileStream(filename, FileMode.CreateNew, FileAccess.Write)) {
 // Write out the input stream
 while (bytesRead > 0) {
 fs.Write(buffer, 0, bytesRead);
 bytesRead = stream.Read(buffer, 0, bufferLength);
 }
 fs.Close();
 }
 stream.Close();
 }
}

Castle Windsor – WCF Endpoint Configuration

Mon, 01 Feb 2010 00:00:00 +0800

 const int maxSize = 52428800;
var binding = new BasicHttpBinding();
binding.Security.Mode = BasicHttpSecurityMode.TransportCredentialOnly;
binding.Security.Transport.ClientCredentialType = HttpClientCredentialType.Windows;
binding.Security.Transport.ProxyCredentialType = HttpProxyCredentialType.None;
binding.Security.Message.ClientCredentialType = BasicHttpMessageCredentialType.UserName;
binding.Security.Message.AlgorithmSuite = SecurityAlgorithmSuite.Default;
binding.MaxReceivedMessageSize = 1000000;
binding.CloseTimeout = new TimeSpan(0, 1, 0);
binding.OpenTimeout = new TimeSpan(0,1,0);
binding.ReceiveTimeout = new TimeSpan(0,10,0);
binding.SendTimeout = new TimeSpan(0,1,0);
binding.AllowCookies = false;
binding.BypassProxyOnLocal = false;
binding.HostNameComparisonMode = HostNameComparisonMode.StrongWildcard;
binding.MaxBufferSize = maxSize;
binding.MaxBufferPoolSize = maxSize;
binding.MaxReceivedMessageSize = maxSize;
binding.MessageEncoding = WSMessageEncoding.Mtom;
binding.TextEncoding = Encoding.UTF8;
binding.TransferMode = TransferMode.Buffered;
binding.UseDefaultWebProxy = true;
binding.ReaderQuotas.MaxDepth = 32;
binding.ReaderQuotas.MaxStringContentLength = maxSize;
binding.ReaderQuotas.MaxArrayLength = maxSize;
binding.ReaderQuotas.MaxBytesPerRead = maxSize;
binding.ReaderQuotas.MaxNameTableCharCount = maxSize;
container = new IocContainer(LifestyleType.Transient);
container.AddFacility<WcfFacility>().Register(
Component
.For<ISharePointFacadeService>()
.Named("DmsGateway")
.ActAs(
new DefaultClientModel()
{
Endpoint = WcfEndpoint
.BoundTo(binding)
.At("http://localhost/SharepointFacade/DMSService.svc/mex")
}));

ConvertPropertiesAndValuesToHashtable Extension method

Wed, 27 Jan 2010 00:00:00 +0800

public static class Extensions { public static Hashtable ConvertPropertiesAndValuesToHashtable(this object obj) { var ht = new Hashtable();
 // get all public static properties of obj type
var propertyInfos = obj.GetType().GetProperties().Where(a=>a.MemberType.Equals(MemberTypes.Property)).ToArray();
// sort properties by name
Array.Sort(propertyInfos, (propertyInfo1, propertyInfo2) => propertyInfo1.Name.CompareTo(propertyInfo2.Name));
// write property names
foreach (PropertyInfo propertyInfo in propertyInfos)
{
ht.Add(propertyInfo.Name, propertyInfo.GetValue(obj, BindingFlags.Public, null, null, CultureInfo.CurrentCulture));
}
return ht;
}
}
Tests

using System; using System.Collections; using System.Globalization; using System.Linq; using System.Reflection;

WCF Tracing

Fri, 13 Nov 2009 00:00:00 +0800

In the web.config in between the tags

<system.diagnostics>
<sources>
<source name=“System.ServiceModel” switchValue=“Information, ActivityTracing” propagateActivity=“true”>
<listeners>
<add name=“traceListener” type=“System.Diagnostics.XmlWriterTraceListener” initializeData= “WCFTrace.svclog” />
</listeners>
</source>
</sources>
<trace autoflush=“true” />
</system.diagnostics>

RandomHelper

Mon, 27 Apr 2009 00:00:00 +0800


using System;
using System.Collections.Generic;
using System.Linq;
using System.Reflection;
using System.Text;

namespace TestHelpers
{
 public static class RandomHelper

 {
 /* This method can be used to fill all public properties of an object with random values depending on their type. CAUTION: it does not fill attributes that end with 'ID' or attributes which are called 'pk'. They have to be filled manually.*/

 private static readonly Random randomSeed = new Random();
 /* Generates a random string with the given length*/


 public static T FillPropertiesWithRandomValues<T>(bool fillBaseObjects)
 {
 return CreateItem<T>(true);
 }

 public static T FillPropertiesWithRandomValues<T>()
 {
 return CreateItem<T>(false);
 }


 private static T CreateItem<T>(bool fillBaseObjects)
 {
 Type type = typeof (T);
 var item = (T) Activator.CreateInstance(typeof (T));

 while (type != null)
 {
 PropertyInfo[] infos =
 type.GetProperties(BindingFlags.Public | BindingFlags.Instance | BindingFlags.DeclaredOnly);

 if (infos.Length == 0)
 {
 if (item.GetType().BaseType != null)
 {
 infos =
 item.GetType().BaseType.GetProperties(BindingFlags.Public | BindingFlags.Instance |
 BindingFlags.DeclaredOnly);
 }
 }


 foreach (PropertyInfo info in infos)
 {
 Type infoType = info.PropertyType;

 Type nullableType = null;


 if (infoType.IsGenericType && infoType.GetGenericTypeDefinition().Equals(typeof (Nullable<>)))
 {
 nullableType = ExtractTypeFromNullable(infoType);
 }


 if (!info.CanWrite) continue;
 if (infoType.Equals(typeof (DateTime)) || infoType.Equals(typeof (DateTime?)))
 {
 info.SetValue(item, RandomDateTime(DateTime.Now, new DateTime(3000, 01, 01)), null);
 }

 else if (infoType.Equals(typeof (string)))
 {
 info.SetValue(item, info.Name + "_" + RandomString(20, false), null);
 }

 else if ((infoType.Equals(typeof (long)) || infoType.Equals(typeof (double)) ||
 infoType.Equals(typeof (int)) || infoType.Equals(typeof (long)) ||
 infoType.Equals(typeof (int))) && !info.Name.ToLower().EndsWith("id") &&
 !info.Name.ToLower().Equals("pk"))
 {
 info.SetValue(item, RandomNumber(0, 999999), null);
 }

 else if ((infoType.Equals(typeof (long?)) || infoType.Equals(typeof (double?)) ||
 infoType.Equals(typeof (int?)) || infoType.Equals(typeof (long?)) ||
 infoType.Equals(typeof (int?))) && !info.Name.ToLower().EndsWith("id") &&
 !info.Name.ToLower().Equals("pk"))
 {
 Type genericType = info.PropertyType.GetGenericArguments()[0];

 info.SetValue(item, Convert.ChangeType(RandomNumber(0, 999999), genericType), null);
 }

 else if (infoType.Equals(typeof (decimal)))
 {
 info.SetValue(item, 1m, null);
 }
 else if (infoType.Equals(typeof (bool)))
 {
 info.SetValue(item, true, null);
 }

 else if (infoType.Equals(typeof (Guid)))
 {
 info.SetValue(item, Guid.NewGuid(), null);
 }

 else if (infoType.Equals(typeof (Enum)))
 {
 Array values = Enum.GetValues(infoType);

 List<object> list = values.Cast<object>().ToList();

 object val = list[new Random().Next(0, list.Count)];

 info.SetValue(item, val, null);
 }

 else if (((infoType.BaseType != null) && (infoType.BaseType.Equals(typeof (Enum)))))
 {
 Array values = Enum.GetValues(infoType);

 List<object> list = values.Cast<object>().ToList();

 object val = list[new Random().Next(0, list.Count)];

 info.SetValue(item, val, null);
 }

 else if (nullableType != null)
 {
 if (nullableType.BaseType.Equals(typeof (Enum)))
 {
 Array values = Enum.GetValues(nullableType);

 List<object> list = values.Cast<object>().ToList();

 object val = list[new Random().Next(0, list.Count)];

 info.SetValue(item, val, null);
 }
 }

 else if (infoType.IsArray)
 {
 Console.WriteLine("Object contains array of objects need to fill these");
 MethodInfo sm = info.GetSetMethod(true);
 if (sm.ReturnType.IsArray)
 {
 object arrayObject = sm.Invoke(item, null);
 foreach (object element in (Array) arrayObject)
 {
 foreach (PropertyInfo arrayObjPinfo in element.GetType().GetProperties())
 {
 Console.WriteLine(arrayObjPinfo.Name + ":" +
 arrayObjPinfo.GetGetMethod().Invoke(element, null));
 }
 }
 }
 }
 }

 if (!fillBaseObjects)
 break;

 type = type.BaseType;
 }

 return item;
 }

 ///<summary>Indentify and extracting type from Nullable Type</summary>
 public static Type ExtractTypeFromNullable(Type type)

 {
 if (type.IsGenericType && type.GetGenericTypeDefinition() == typeof (Nullable<>))

 {
 PropertyInfo valueProp = type.GetProperty("Value");

 return valueProp.PropertyType;
 }

 else

 {
 return null;
 }
 }


 public static string RandomString(int size, bool lowerCase)

 {
 var randomString = new StringBuilder(size);


 /* Ascii start position (65 = A / 97 = a)*/

 int start = lowerCase ? 97 : 65;


 /* Add random chars*/

 for (int i = 0; i < size; i++)

 {
 randomString.Append((char) ((26*randomSeed.NextDouble()) + start));
 }


 return randomString.ToString();
 }


 public static int RandomNumber(int minimal, int maximal)

 {
 return randomSeed.Next(minimal, maximal);
 }


 /* Returns a random boolean value*/


 public static bool RandomBool()

 {
 return randomSeed.NextDouble() > 0.5;
 }

 /* Returns a random color*/

 public static DateTime RandomDateTime(DateTime min, DateTime max)

 {
 if (max <= min)

 {
 const string message = "Max must be greater than min.";

 throw new ArgumentException(message);
 }

 long minTicks = min.Ticks;
 long maxTicks = max.Ticks;
 double rn = ((Convert.ToDouble(maxTicks) - Convert.ToDouble(minTicks))*randomSeed.NextDouble()) +
 Convert.ToDouble(minTicks);

 return new DateTime(Convert.ToInt64(rn));
 }
 }
}